So Yoast SEO was running REALLY slow, like 2 minutes slow.
Turns out it renders page content via wp_ajax and then parses this content via javascript. This is very problematic if a certain shortcode renders lots of content on the frontend (in my case it was a job board/map search shortcode thing).
A quick fix is to just kill the shortcode parsing for Yoast with this code in functions.php
add_action( 'wp_ajax_wpseo_filter_shortcodes', function(){
wp_die( wp_json_encode( array() ) );
}, 1 );