The Github Repo with the code associated to this text
VLC is a media player. It's purpose is to play videos or audio recordings or just images. It's quite a capable piece of software that's even able to show subtitles. This has inevitably brought the following thought into the mind of any normal person: would it be possible to create a video file that runs a video not as a video but sequence of subtitles?
We are here to answer the age-long question that was on everybody's mind: can you transform a video into subtitles?
Ok so a quick lesson on subtitles in videos. A very common format for subtitles is ".ass". Yes. Some software engineers were fully aware of what they were doing when they chose the acronym "Advanced SubStation Alpha". It's a format for subtitles that gives the creator incredible power over them. You can chose the font, the color, the position and even make the text move and animate. It all works with the following format (it's best explained by seeing the actual .ass program):
[Script]
Title: test
ScriptType: v4.00+
WrapStyle: 0
PlayResX: 1920
PlayResY: 1080
[V4+ Styles]
Format: Name, Fontname, Fontsize, PrimaryColour, SecondaryColour, OutlineColour, BackColour, Bold, Italic, Underline, StrikeOut, ScaleX, ScaleY, Spacing, Angle, BorderStyle, Outline, Shadow, Alignment, MarginL, MarginR, MarginV, Encoding
Style: Default,Arial,60,&H00FFFFFF,&H000000FF,&H00000000,&H80000000,0,0,0,0,100,100,0,0,1,2,2,2,10,10,10,1
[Events]
Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
Dialogue: 0,0:00:00.00,0:00:03.00,Default,,0,0,0,,{\move(100,540,1820,540)}hello this moves across
Dialogue: 0,0:00:03.00,0:00:06.00,Default,,0,0,0,,{\pos(960,200)\c&H0000FF&}red text at top
Dialogue: 0,0:00:06.00,0:00:09.00,Default,,0,0,0,,{\pos(960,540)\t(0,3000,\frz360)}spinning text
This kind of program is easy to edit and can be imbedded into a video with the following command:
ffmpeg -i input.mkv -i test.ass -c copy -c:s ass output.mkv
This is great, but to transform any video into subtitles, we'd need a way to have the subtitles show the equivalent of pixels rather than just pieces of text moving around. The problem of turning images into pieces of text is a solved one. you just need to look up "image to ascii" to find plenty of examples.
This program uses a simpler approach:
Reading pixel values from a frame in a mkv file is relatively easy (by asking an AI) in cpp. You can open a mkv file with a cpp script and then read rgb color values at different (x, y) positions in a frame. You can then make a script that writes new subtitle "pixels" for the different positions and colors by appending Dialogue: 0,0:00:08.96,0:00:08.98,Pixel,,0,0,0,,{\pos(1432,1060)\1c&H526152&}█ with the correct x, y and color values to create a pixel in the final image.
Transforming this into a video is trivial: you just add the right start and end times for each subtitle and have the program translate multiple frames with time.
So it turns out that neither ffmpeg nor vlc appreciate having 1920x1080 or 2 073 600 individual subtitles blasted at them every single frame. I'm not sure I understand why. It feels to me that running over one million subtitles per frame should be the intended purpose of multimedia viewing software. The solution is to decrease the resolution of the subtitle video (I have found that a pixel width of 20 works quite quite well). We must also decrease the frame rate for the subtitle video to be manageable for the viewing software (vlc) to handle. A video that I translated into subtitles ended up with a 5,239,095 line .ass file containing every pixel! that's HUGE.
When I add the subtitles to the video, the video passes from a size of 19KB to 439 MB. That's an addition of about 438MB to the file, having multiplied its size by a factor of about 460 million. The original video was simply a black background so the optimizations must have reduced it to a large extent but we still have 460 million times more subtitle than video in this mkv file. Regardless, the results look excellent. We can tweak the width and heights to have the best viewing experience and I have found that mpv is far far better at showing this ridiculous amount of subtitles than vlc is. It's nearly as though mpv were made for viewing videos the correct way: in the subtitles. This is great for rick rolling people: you send them a normal video and then tell them to turn their subtitles on.
This clip is a video entirely comprised within the captions. The video without the captions is a simple black background
If for some absurd reason, you wish to interact with this abomination of code and this horrendous use of time and human effort, you can set this up and run your own videos in subtitles (or edit the code to your heart's desire) here.