During Wednesday/Thursday 12/13th February it was noticed that the Callback leads generated from Facebook ads (known by Facebook as “leadgens”) had dropped off to zero. After investigation it was discovered that the Facebook app (“Adrian Flux Webhooks”) that was developed to send the leads to the callback service had disappeared completely.
The Facebook apps are tied to a Facebook account and then various access tokens are generated against this account. 99% of issues are to do with access tokens suddenly not working without any warning.
The Sterling main brand site was unavailable for a period of around 1 hour on the evening of Friday 18th, staring around 7:45PM. This was caused by an issue in the external hosting service for one of the site services. This was totally unrelated to the previous outage.
We were immediately notified of the outage by the now correctly configured monitoring service and was immediately investigated. It was found that one of the sites dependent microservices was unreachable which was causing requests to the main brand site to timeout.
A new type of Facebook lead generation (leadgen) form is being configured by Optimyzd that will allow scheduled style callbacks to be requested directly from Facebook adverts.
This will require a new handler in the Callback service to be configured that will deal with the new data fields that will be sent with the leadgen. The handler will build upon the existing instant callback handler.
For reasons unknown, the Facebook-provided testing tools to not work correctly with this type of form.
The Sterling main brand site was unavailable for a period of around 2 hours on the morning of Friday 18th. This was caused by a change to the proxy configuration which was incorrectly deployed by an external automated process. This automated push should have been anticipated (by Hut) and handled in the correct way.
As soon as we were aware of the issue, a fix was immediately created and pushed to the proxy which resolved the issues fully.
The new Bikesure dealer referral capture was added to the Bikesure live proxy. This will offer conversion and anti-spam improvements over the existing ones.
The Adrian Flux version was rerouted to a new URL https://www.adrianflux.co.uk/referral-scheme/ to be consistent with the Bikesure brand. This consistency will be important when we get all dealer-style referral management and workflows into a single application.
Work will be ongoing over the next coming months (once fully scoped) to get to the beta version of the referral system.
The new EPA version, 1.11 got pushed live to the Van EPA instance. This adds the callback functionality when certain quote criteria are met.
The plan is to canary test with this instance for a while and then push the version out to the other (Car, Bike etc. instances.)
The new Flux job application form / management tool is in the final phase of the development. Target is to have the Beta version of the Job Application Management (JAM) tool available for testing at the end of the month.
The new front end form, in the meantime will be connected to the existing management tool. This will allow the new and improved front-end to be available before the testing of the new JAM tool has been completed.
The new proxy architecture that was deployd for Adrian Flux and Bikesure has now been configured and for use with Sterling. Previously all requests were going to the content instance (sterling-wordpress) but are now being routed through a dedicated proxy of its own.
This will improve performance and stability as the traffic can be managed identically to the other brands, taking advantage of a shared firewall rule set.
Also, the requests are being handled at edge by the Worker process which means that edge feature such as content rewrite (for common content strings) and edge redirects will be available on the Sterling domains too.
The updated Victorian homes WPE based content site was added to the live proxy.
The exchange handler was also updated to work with the improved forms on the new content site. The change should be transparent to both the end-users and the mailing list / Instiller automation.
Currently the Policy documents that are available for download to Adrian Flux, Bikesure are Sterling customers are handled via the (now legacy) customer portal. Since the migration to the WordPress based customer portal, the existing policy document is still being handled by the existing customer portal. The documents themselves are still being managed in the existing portal backed. The documents themselves are being delivered to the corresponding Brand sites via an API in the legacy portal.
The callback service 2.12 was released to live. This release added in the necessary changes to allow Sterling Northern Ireland callbacks to be processed as Sterling callbacks. This was a follow-on from the Facebook permissions work.
The Facebook Northern Ireland page is going to start running callback related lead ads. This is exactly the same functionality that has been running on the via the Sterling Facebook page. The forms are hooked into the callback service via a custom Facebook application that was developed in order to capture and process the requests. The internals of this are quite complex (as it involves the Facebook API) but it does allow Maze to simply add new leadgen adverts, and they will be processed by the service.
An audit and branch cleanup was completed on referral-system-angular. This completed all the necessary pull requests into live and removed dead and stale branches.
A build script was also added in order to push different angular build configurations in the Cloudflare Page deployment. This will allow API changes to be tested on a separate front-end instance. The build script ande deployment settings can also be reused for other Angular/Cloudflare projects.
The callback service 2.11 was released to the live Adrian Flux site. This was a patch release that fixed the issue of html entities appearing in the callback entries.
This corrected entries such as https%3A%2F%2Fquotes.af-test.co.uk%2Fcar to https://quotes.af-test.co.uk/car.
As an update to the previous work, it was observed that the blog urls such as https://www.adrianflux.co.uk/blog/2023/06/fuelling-around-podcast-greg-james.html/ were being rendered with the trailing slash. This was caused by the custom code that was deployed to add the trailing slashes being added to the single post pages as well. Normally this would not be an issue, but in the case of the Fluxposure blog the canonical urls were configured to end with a slash.
The taxonomy (categories, tags, series) pages on Fluxposure were timing out. This was also obseved in the proxy logs, with many H12 timeout errors appearing.
After investigation, this was caused by the category links in the site theme somehow being configured to not have traling slashes. This would cause a redirect to remove the slash if one was present; i.e. https://www.adrianflux.co.uk/blog/tag/3d-printing/ would be redirected to https://www.adrianflux.co.uk/blog/tag/3d-printing. Since all the URLs going through the proxy get normalized to add a traling slash (at edge), this was causing an infinite redirtect loop between the blog site and Cloudflare edge.
A project is in development to resolve the state of a department given a set of rules. A rule is defined for a
department with a certain state e.g. open. For example, between 09:00am and 05:30pm this department is in state open.
Rules can be more granular to account for specific days, dates and sources. A department has a default state as a
fallback when there are no rules applied.
Minor update to flux-callback-services to address some issues found in the new EPA updates.
Meta fields, web_lastname, source, external_referer and internal_referer fields were cleaned prior to saving in order to prevent entity encoding characters such as 124%2F273%2FZ167%2FWEB being saved to callback fields.
An issue was identified with Instiller receiving workflow pushes for EPA incompletes. After other more critical priorities were addressed, the issue was looked into for resolution.
A broekn version of the proxy config was inadvertently pushed to the main proxy branch on 12th June 2023 at around 8:30 AM. This should have gone to the test versions only. This caused the sites and applications on www.adrianflux.co.uk and www.bikesure.co.uk to go down for around 30 seconds.
To mitigate against any future incidents such as this, auto deployments have been disabled on the live versions of these proxies.
It was brought to our attention that organic traffic had dropped off significantly from on some flux sites on Thursday (2023-06-08). The drop-off started to occur around the time of the proxy switchover (2023-05-25) and was declining at a constant rate since then.
A proxy update was applied the morning of 6th June. This was to enable Yoast to better handle redirects. In addition, the Cloudflare worker was changed to handle the end-slash redirects.
The Bikesure dealers directory had an error returned when attempting to lookup a dealer by postcode. This was caused by the live proxy being incorrectly configured to point to the test dealers system.
This is currently not returning any dealers as it’s undergoing some maintenance work to try to use the newer version of the geo service.
The live proxy was correctly pointed to the live dealers application, and normal functionality was restored.
Minor updates to django-bulk-redirects as used by flux-redirects-service.
Pre-save validation added to provide error message about duplicate direct paths for a brand. Previously the user would have seen an “Internal Server Error” on attempted save of a duplicate. New status codes added to match Yoast redirects, These are: 307 - Temporary 410 - Content deleted 451 - Content unavailable for legal reasons The eventual goal being Yoast redirects will sync and push up to edge so WP admins could exclusively manage them in there for the sites but the redirect would still happen at edge.
The exchange-services suffered an outage this morning that resulted in a downtime of a few hours. This update will provide details of the reasons for the incident, and it’s resolution.
The exchange service recently has been updated to run on Amazon Linux 2 in the AWS Elastic Beanstalk (EB) container service. This upgrade was rolled out a couple of weeks ago and has been running successfully since the upgrade. As part of the upgrade the opportunity was taken to add a new load balancer for all the new and updated EB services.