Flow rules will work stateless, so them can’t manage on connection layer correctly. You can make rule like this:
drop
ethertype ipv4
and ipprotocol 6
and chr tcp_syn
and ipsrc 192.168.98.21/32
and ipdest 192.168.98.100/32
and not dport 22
;
This rule will drop any packets from 192.168.98.21 to 192.168.98.100 with destination port is not a 22. Place ip before latest rule for accept. This rule will not affect to packets in reverse side directly but can broke another network services.
I edit this example, now it affects only TCP packets that start new connection, them don`t affect to UDP, ICMP or another protocols that can be inside IPv4/v6. It still can broke services that you can try to filter with it like FTP, SIP or similar, so be careful.
I say it one more time - this is stateless filtering, pay attention to this!