You know Duff Goldman for his Food Network show “Ace of Cakes,” filmed at his Baltimore-based Charm City Cakes, or his appearances on other cooking shows like “Cake Masters.” But when the celebrity baker stopped by Milk Street Radio, he spoke about a lot more than just frosting and cake batter.

Goldman started out as a graffiti artist, for one. He also never set out to open his own cake shop; he started baking cakes in his apartment to pay the bills while he was trying to be rock star. As he says, “any day now.” Check out the excerpts from the full interview below, and listen here on Milk Street Radio for more.

On his former life as a graffiti artist

There was a yard in Braintree, Massachusetts—I grew up on Cape Cod in Sandwich—so we'd go up to Braintree, break in and paint trains, and hopefully be able to leave by the same hole in the fence that we made, because if you're leaving any other way, that means you're in the back of the cop car and that sucks.

Well, sometime they give you a choice. You can either get arrested or—this happened one time and I'll never do it again but—you can empty out all the paint that you have onto yourself, including your face and your hair and everything. You have to empty out all the paints that you have on yourself, and then they'll let you leave.

On the half-baked start of his now-famous bakery

I moved back to Baltimore so I could be in a band and I got a job as a personal chef. That was going to be my sort of step away from working in restaurants—I was kind of done. I was burnt out. So I was a personal chef, playing music, and then once my band was really getting big enough that we were getting gigs, I quit that job and to pay the rent, I started making cakes in my apartment. I built a website. I stole a bunch of pictures from other awesome cake decorators, putting them on my website, and then every time somebody would call and be like, hey, can I get that—that yellow cake with the brown stripe on it with the blue flower—yeah, sure. Then I'd make it and then I’d take the picture down that I stolen, and put up the picture of the one that I actually made.

On the similarities between playing in a band and cooking in a kitchen

I think that in most kitchens you can find musicians. There’s the same sort of energy, vibration, whatever it is, that draws people to kitchens that draws them to not just music but being in bands. Because being in a band, you really have to coexist with other creative people, and creative people sometimes are really stubborn, because they have a vision.

One of his proudest moments in baking

I think my coolest discovery was when I figured out how to create hyper realistic wood grain on fondant. That was a big day. I know it sounds probably pretty random, but that was one of my proudest achievements figuring out how to do it.

One of the oddest requests he’s ever gotten

A nurse's convention wanted a cake of the dummy that they use—that is, a model for all of the different kinds of bedsores that you can get. They brought us the model and it was gross. They were like, that's what we want the cake to look like: a bedsore-ridden, armless torso. The people in the medical field have the weirdest senses of humor because I think they face mortality every day, and they just have to laugh.

This interview has been edited for clarity.