Cannot find Working Environment/Blocked messages | NetApp | Page 1

graceful hinge May 30, 2023, 3:11 PM

#

Starting on 5/20 in the region we've started seeing "Cannot find working environment" and "Error: code: 403, message: Blocked" messages when executing our Terraform plans. We're managing 9 ONTAP clusters in the GCP us-central1 region and have been using this code successfully for over a year. I've tried changing to the latest version of the Terraform provider (and other older ones) with no luck.

An odd thing is if I try targeted applies (only managing a few clusters and volumes) then I don't get these errors. One theory I have is that I'm being rate limited by the Cloud Manager APIs. I've tried setting the TFE_PARALLELISM to "1" but that also hasn't helped. I'm stumped so looking for any suggestions.

thanks!
Paul

proud cairn May 30, 2023, 5:28 PM

#

Hi, do you see errors on every resources you manage? or some of them?

graceful hinge May 30, 2023, 7:14 PM

#

Which resources it errors on varies. It'll error on 2 resources on one run. Then on the next run it'll error on 14 completely different resources. I've also taken a few resources it does error on and ran targeted plans against them and they successfully plan.

proud cairn May 30, 2023, 10:25 PM

#

There isn't a lot changes going on with the provider so I don't think it's the provider issue. Are you using terraform cloud. According to https://developer.hashicorp.com/terraform/cloud-docs/workspaces/variables: "Terraform Cloud Agents do not support TFE_PARALLELISM, but you can specify flags as environment variables directly via TF_CLI_ARGS_name. In these cases, use TF_CLI_ARGS_plan="-parallelism=<N>" or TF_CLI_ARGS_apply="-parallelism=<N>" instead."

graceful hinge May 31, 2023, 3:22 AM

#

We're using Terraform Enterprise, not TFC.

graceful hinge May 31, 2023, 3:08 PM

#

When I set the debug level to trace I see the provider successfully fetching Working Environments for 14 seconds and then all of the logs suddenly change to "getWorkingEnvironmentInfo request failed: Blocked" and all requests from there on get the blocked message

proud cairn May 31, 2023, 5:34 PM

#

are you able to share the trace?

graceful hinge May 31, 2023, 9:03 PM

#

I dont want to upload log files to a public discord, I've blanked out the server names (in green) and the working environment IDs (in red) in this log where it shows requests going from successful to mostly blocks

#

https://kb.netapp.com/Cloud/BlueXP/Cloud_Manager/Unable_to_discover_WKE_and_deploy_new_CVO_due_to__API_limit_on_Azure_side I wonder if we're running into something like this but I'm not seeing any messages that would confirm it

NetApp Knowledge Base

Unable to discover and deploy new CVO due to API limit in Azure

proud cairn May 31, 2023, 9:28 PM

#

And it errors out at different steps everytime?

graceful hinge Jun 1, 2023, 12:34 PM

#

it has suddenly started working, I'll see if that continues. I'm suspicious that me experiencing these issues perfectly correlates to this maintenance

#

although thats CVS so perhaps its not related

stone ibex Jun 1, 2023, 4:04 PM

#

Thanks for the confirmation. Yes, there was an adjustment on BlueXP side.

graceful hinge Jun 2, 2023, 2:01 PM

#

Were there any changes that would affect Cloud Manager and Cloud Volumes ONTAP? Our errors have now resolved without us making any changes but I'd like to have some idea of root cause. Thanks

proud cairn Jun 5, 2023, 9:41 PM

#

The cloudmanager responsed there was indeed a rate limiting issue, they increased the limit for traffic coming from Terraform to allow more API requests.

graceful hinge Jun 6, 2023, 12:58 PM

#

great, thanks for the update wenjun

#Cannot find Working Environment/Blocked messages