Is there a way to convert a .rofl file to .mp4 or .avi?

No, there is no way to directly convert .rofl files to any other video format. This is because they are not video files themselves - they just contain "spectator data" so that the game client knows what actions happened at what times, which allows it to replay those actions in sequence for your viewing pleasure.

If you want to make an actual video recording, you'll need some kind of screen-recording software. There's a whole bunch of options - OBS, Plays.TV, and Fraps are probably the most commonly used ones, but there's tons more too. These will record to actual video files (.avi or .mp4, usually) that you can then convert or use as you please.

As mentioned in another answer, you can use the built-in recorder as well, if you just want to save some clips. The result of using the built-in recorder is .webm file, which is a lightweight, web-friendly video file. After testing it, I can't see any hard limit on the amount of time that a single highlight can take (I was able to take 20 minutes of video without any problems). However, the quality of that recording isn't the greatest. I don't have a tool to take exact measurements, but I would estimate the resolution is about 1280x720. There are a lot of compression artifacts, though, so it doesn't look very crisp or high definition.

"replay" files for games are generally just instructions to the game engine, not actual video.

So if you want to convert them to an actual video you will most likely need to replay them in the game while recording the output of the game with a screen-recording soloution.