During Wednesday/Thursday 12/13th February it was noticed that the Callback leads generated from Facebook ads (known by Facebook as “leadgens”) had dropped off to zero. After investigation it was discovered that the Facebook app (“Adrian Flux Webhooks”) that was developed to send the leads to the callback service had disappeared completely.
The Facebook apps are tied to a Facebook account and then various access tokens are generated against this account. 99% of issues are to do with access tokens suddenly not working without any warning. In this case however it seemed more severe as the apps were not even tied to the original account. 2FA auth was also not working at all between the Facebook account and dev account. In this case:
- A new Facebook account was created
- A new Facebook Developer was created
- Business Manager permissions were debugged and added to the new account
- A new SIM card was purchased for the 2FA as the original number was no longer receiving any 2FA messages
- A new Facebook app (“Webhooks”) was created as even though the existing app was now visible no access tokens could be generated
- New User and Page access tokens were generated for use in the app. A significant change in the Facebook token setup now seems to be each brand must have its own tokens.
- Business manager permissions were debugged and added at a CRM level
Whilst this all seems trivial, it in fact represents 2-3 days work (including over weekend). There is a lot of out of date or no existent documentation for Facebook API, or in some cases it’s simply broken. An example of this was critically the Lead Ads testing tool. There is a lot of frustration online about this now not working for people.
A review was done on how the Webhook app could and should work and it was decided to change it up a bit. This was for:
- Handling the multi access token paradigm
- Better visibility for incoming leads from Facebook
- Monitoring opportunities for incoming leads
- Monitoring opportunities for access tokens
- Leveraging AWS serverless infrastructure for more rapid and consistent development
- Development of a new report along with a micro-frontend for incoming leads
Development of this and the micro-frontend was done over a few days, along with automated and on-demand reports. The client (Optimized) was very happy with the results, especially with the front-end for realtime visibility into incoming leads.
From a Hut Forty Two perspective this small body of work had some very positive outcomes:
The receive and log to callback process was fully managed in an AWS state machine in front of an API gateway resource. This cut the amount of custom code to write to very little. Going forward, this will be the blueprint for other callback handling with the eventual goal of moving all the handlers to their own state machine/step functions. This will make these processes very cheap, reliable and consistent. It will cut down code and server maintenance to almost nil. The micro-frontend was also developed in parallel with the small frontend needed for the new JAF system. This will very much be the architecture going forward: API routes in API gateway, State Machines for object workflow management with Lambdas for custom code execution. More on that to come in the immediate future.