Issues with the refresh endpoint endlessly redirecting after signin

[HudsonAkridge]

We're using v2.1.0 (vanilla, nothing fancy/special). Occasionally, a user will get stuck in this redirect loop. I suspect it's when the JWT expires, and a lazy evaluation happens. Not all the time, but sometimes, it will put the browser into this constant back and forth redirect ping thing that keeps showing the message/image displayed below. If you look at the XHR request stuff, you'll see that it's just endlessly redirecting from the /refresh endpoint, to the cognito /auth and /login endpoints. I can post the XHR's (once I sanitize them), if needed.

This might be due to a refactor done recently for this version that has the JWT refreshed lazily perhaps?

I've tried with the JWT expiration set to 1 hour, 12 hours, 24 hours, and 5 minutes and the same thing. It just changes how fast the cycles go before this issue manifests/gets recreated.

For the end user, there's no solution other than clearing cookies to correct this problem. Once they sign in again, the problem goes away for several refreshes, some unknown number (can't recreate it consistently, it appears to be a timing thing) before it hits again.

I'm going to try reverting back to a previous version of the app (v2.0.19) to see if this has any positive impact.

[ottokruse]

Thanks for reporting this.

What are your values for parameters enableSPAmode and cookieSettings?

To debug, probably best to set loglevel to DEBUG for a bit, if that's acceptable to you (logs all requests, include sensitive data). That will show the responses generated by the Lambda@Edge functions––would be good to see if the checkAuth actually did set the nonce, and whether that nonce was sent by the browser to the refresh endpoint. You can also check that in the network panel.

I can post the XHR's (once I sanitize them), if needed.

That would be great.

[HudsonAkridge]

EnableSPAmode = true
CookieSettings = { "idToken": null, "accessToken": null, "refreshToken": null, "nonce": null }

I do have some debug logs that should have been captured. Just for the /refreshauth lambda though. Let me see if I can get those (this has been a problem for the last week, we've been trying to figure out the cause).

Also, it'll take a few for me to get those XHR's, but will do. I'll post here with additional data.

So far, for what it's worth, I rolled back to v2.0.19 and haven't had this issue show up in the last 24 hours. Except the initial time when I rolled it back and changed the JWT/ID token expiration from 24 hours back to 60 minutes. But, since then, no issues reported.

[ottokruse]

Those settings are vanilla and should work.
You mentioned "Occasionally" which is interesting -- any clue as to the precise scenarios where this does or does not occur?

[HudsonAkridge]

@ottokruse
Re: Scenarios. Unfortunately there's not an easy way to recreate it. What I do know are the following based on pulling a few different levers:

1. It happens no matter what the JWT/ID expiration is set to in the Cognito client
2. If the JWT/ID token expiration is set to longer (e.g. 12 hours/720 minutes), it takes much longer to recreate the issue, but it always seems to happen
3. It appears to happen whenever the JWT/ID Token expires, when the user hasn't hit the site in a while. e.g. they were signed in yesterday morning, the token lasted 12 hours, and now they try and refresh the site the next morning, that seems to trigger it more often than other patterns.
4. If they're using the site continually, it doesn't seem to happen, it appears to mostly be after letting it sit for a bit past the JWT/ID token expiration.

[HudsonAkridge]

@ottokruse
This took a lot longer than I expected to sanitize the .har file 🙂 but finally got it done. I've redacted anything that could identify us, but I don't think anything I've removed would make it more difficult for you to see the issue.

LMK if I'm wrong about that and you need some bit of data that was redacted. If it's not an identifying piece of data for the user or our app, I'd be happy to add it back.

I had to change the type to .txt for it to upload to here so you'll have to change it back to .har 🙂
redirects-issue-redacted.txt

[HudsonAkridge]

The largest issues with the redirect loop are:

1. A normal user is unlikely to notice this and it'll just keep redirecting in the background forever. Hundreds/thousands of times until they stop the cycle
2. The Sign-In button is broken and won't actually work to let them sign back in.
3. Clearing cookies is the only way to solve this right now we've found, which is obviously not ideal.

We've got a handful of power-users that are going to test our app with v2.0.19 over the weekend. I'll let you know if we see the same problem there. But so far, no issues. Only with v2.1.0 (which could make sense, as a lot changed with the refreshauth logic in there it looks like)

[ottokruse]

Thanks that was very helpful.

I can see the redirect loop, going back and forth to /refreshauth, until chrome throws net::ERR_TOO_MANY_REDIRECTS. That then leads to another trip to /refreshauth but this time without cookies. That would then show the error page you pasted the screen shot of above.

The root error is the redirect loop. Interestingly, from the HAR file the refresh seems to work okay:

• User is redirected to /refreshauth (apparently because JWT expired)
• User is redirected back to /, with new JWTs in the response.

All good, but then checkAuth decides again to send the user to /refreshauth.

Need to look closer into the reason for that. Best next step is to enable info logging, that will show the exact reason why checkAuth thinks the JWT is invalid in the logs:

cloudfront-authorization-at-edge/src/lambda-edge/check-auth/index.ts

Line 40 in 3a79edd

CONFIG.logger.info("Access denied:", err);

Have a look at what the error says exactly?

You didn't add the JWTs in the HAR (makes sense) but it would be good to have a look at them and maybe run them through jwt.io and verify with your own eyes that the JWT's in the cookies in the response from /refreshauth are good valid and not expired.

[ottokruse]

The Sign-In button is broken and won't actually work to let them sign back in.

Can you share more details on this too? Maybe there is a clue here. Just tested again on 2.1.0 and for me this button works fine: it signs you out under the hood and redirects to the Cognito Hosted UI to sign in again.
Related: what setting do you have for parameter SignOutUrl (the default value is /signout)

[HudsonAkridge]

The Sign-In button is broken and won't actually work to let them sign back in.

Can you share more details on this too? Maybe there is a clue here. Just tested again on 2.1.0 and for me this button works fine: it signs you out under the hood and redirects to the Cognito Hosted UI to sign in again. Related: what setting do you have for parameter SignOutUrl (the default value is /signout)

The value is set to /signout, as the default. And that's what the Cognito HostedUI has. It shows: https://env.my-site.com/signout as the URL for the sign in button in the cognito UI, and this doesn't work at that point. However, the signout button/URL we have in the app that takes you to the same URL does work if you're not currently dealing with the refresh auth redirect issue.

We are using ReactJS and Amplify here, and I have the amplify config set to use cookies. Which does work, it bypasses the amplify sign in page.

I could turn off amplify from the UI if needed for testing.

After redeploying v2.0.19, we're still seeing the same thing eventually. Just had a lot of users experience this issue over the weekend and report it, it just seems to happen less frequently.

The scenario that caused it was one of our users had their computer update and reboot overnight, and when they opened up the page this morning, it triggered a "sea of redirect problems".

My logging was reverted after the 2.0.19 deploy, so I'll update it to at least the INFO level and see if I can't get you exactly what the errors are saying.

I'll also decode the JWTs from that redacted .har I uploaded and see if the timings for the refresh auth endpoint are not being expired prematurely.

[ottokruse]

My logging was reverted after the 2.0.19 deploy, so I'll update it to at least the INFO level and see if I can't get you exactly what the errors are saying.

Cool.

What cookie settings do you have in your Amplify config? This is a thing that changed in v2.1.0, recommend now to have config like this (to use host only cookies):

cookieStorage: {
      domain: " ", // Use a single space " " for host-only cookies
      expires: null, // null means session cookies
      path: "/",
      secure: true, // for developing on localhost over http: set to false
      sameSite: "lax"
    },

Also might be wise to have lines like this in your web app, to let Amplify do the refresh:

cloudfront-authorization-at-edge/src/cfn-custom-resources/react-app/react-app/src/App.js

Lines 44 to 46 in 3a79edd

	// Schedule check and refresh (when needed) of JWT's every 5 min:
	const i = setInterval(() => Auth.currentSession(), 5 * 60 * 1000);
	return () => clearInterval(i);

[bedaka]

@HudsonAkridge could you finally solve this issue?
As it seems like we're experiencing the same problem I would appreciate any hints how to approach this.

[ottokruse]

Hi @HudsonAkridge ! Did you ever get to the bottom of this? Did the logs reveal anything usefull?

[ottokruse]

Closing for now, can re-open if further info surfaces

[HudsonAkridge]

Hi @ottokruse , long time no chat 🙂

We had some other stuff come up and just got around to giving this another go, thinking maybe something updated in the last 6+ or so months to solve the problem.

Did another deploy with the latest (2.1.5) and ran into the same issue (see attached XHR/Network logs)

So, still not sure what's happening/going on here.

What I'm planning to do tomorrow is:

1. Do a full on vanilla deploy of this stack, without any parameters passed in other than what's strictly necessary
2. Let that create the Cognito pool and all other resources needed
3. Set up a few default/basic users
4. Use the default URIs and everything for doing the auth
5. Drop in our ReactJS app into the site's S3 folder instead of using the demo app
6. Try and resolve whatever CORS/config issues come up as a result of that
7. Test again

The only thing that seems non-vanilla about our setup, is our pre-existing custom configured Cognito pool and how we've got some custom domains set up with Amplify.

So the goal is to remove as many variables from our setup as possible to create a control and see if we can recreate the issue then. It just doesn't make any sense that we're the only ones running into this with (potentially) thousands of consumers of this system.

Thoughts?