I found some old code I wrote last year that does exactly what I need for pathfinding, so I'm making good progress :D

To speed up pathfinding, I'm looking into hierarchial pathfinding. Since some of my scenes are quite large grids, and I'm eventually going to need to pathfind between maps as well, I'd like something that at a general level lets me layer sections into pathfinding to be more efficient. Tiles will be in sections, sections will be in maps. Basic algorithm is as follows:

# start - (tile position, section, map)
# end - (tile position, section, map)
if start.section == end.section:
    return astar(section.grid,start.position,end.position)
if start.map == end.map:
    return astar(map.sections,start.section,end.section)
return astar(map_grid.maps,start.map,end.map)

Sections will be created inside of each map, probably by hand to start with as something I can paint in tiled, including places where sections connect. When astar is run on sections, it will need to both find a path from the start to the end section, but also return paths within each section in order to generate the path through the individual tiles. For optimization, I could preprocess these paths and save them. Another approach would be for the ai to constantly be shifting which level it is searching based on where it is in following the path. So once he reaches the boundary between sections, he does another small astar search on the tiles to get across that section.

The biggest benefit to this, is a huge optimization for when I close off parts of the environment. When you move a rock in front of the path blocking an ai in a room for instance, that ai will know that he can't get out of the room, and not try to pathfind like crazy to find an exit. A smaller benefit is much quicker pathfinding if there are many units trying to travel long distances across the map.

Also map transitions is something that I handle very hackily right now. I still need to fix the path following algorithm, lol.

I've been thinking about ways to use the crawlers that I wrote last week. Some ideas that I had were maze generation (it is out because the mazes are actually pretty lame), tile texturing (the tiles have to be really large for it to make sense), and random entity movement (not sure why this would ever be useful). Then I noticed that all of the negative space paths always touch the outside border, so if I spawned crawlers on the corners of the space I could fill in the pattern every time. So then I decided to try them with scene transitions and I really like the look. It would only work in games with a certain style and the speeds need to be adjusted because the transitions are too long now, but I'm happy with it.