From Madera go east on Highway 145. If you are headed to the north shore of Millerton Lake, watch for the signs at Road 145. For the south shore, turn right on Friant Road, following it around to the town of Friant. There you turn left on Millerton Road. You will see the entrance to the lake almost immediately.
From Fresno go north on Highway 41 to North Friant Road and follow it to the lake. Alternately, stay on Highway 41 until you reach the intersection with Highway 145. Turn right and follow it to Millerton Lake as described above.