He grew up in the central Victorian city, leaving only because of the war, but managed to leave a legacy for many in the city.
The older generation in Bendigo will remember the drive-in cinema with fond memories. In fact, many Bendigonians may have been conceived at the very place!
That cinema was a Houlahan project.
Most of the work was done by his brother Kevin, but Jim was the brains behind the project, his daughter Fran told Melbourne Racing.
"After the war, dad and Kevin started a building business and moved down to Melbourne. Dad wanted to build the drive-in at Bendigo, so they flipped a coin to see who would go to Bendigo and who would stay in Melbourne," Fran said.
"Kevin won or lost the toss – whichever way you want to look at it – so he went to Bendigo and dad stayed in Melbourne."
The drive-in cinema was popular for two decades until a sit-down theatre was built.
But the most visible landmark in Victoria that can be attributed to Houlahan is Calder Park raceway.
Every Bendigo resident that has been to Melbourne – by car or train – has been past Calder Park.
Situated on the northern outskirts of Melbourne, the raceway was built on the back of Houlahan’s racing success.
But still, that was in Houlahan's entrepreneurial days, well before he became a trainer.
"He bought the land for Calder Park after he won an all-up bet on three of Bill Williamson's horses at Flemington one day," Fran explained.
"Dad won something like 30,000 pounds and the block out on the Calder (highway) was the exact amount of money he won.
"He went out and bought the block and he formed the company to develop it and build Calder Park."