Terminate single processing server based on SQS queue condition, but keep/ reinstate autoscaling size at 1

Disclaimer: this is certainly not the best solution for backend workers, I'm just trying to be creative within the given constraints.

First, you'll need to be familiar with SQS's visibility timeout (documentation) for this to work.

Let's assume the following, then I'll walk through how it might work:

We only have 1 request (henceforth "message") coming in at a time (10 per day, hopefully that's feasible)
We can take up to 12 minutes to process a message in the worst-case scenario

Next, some configuration changes:

We configure the ASG to scale desired count to 0 when ApproximateNumberOfMessagesVisible = 0 for 5 minutes.
We configure the ASG to scale desired count to 1 when ApproximateNumberOfMessagesVisible >= 1.
We set all messages VisibilityTimeout to 10 minutes.

A failure scenario might work out like this:

At 0 minutes, 1 message is being processed by the worker. There are now 0 other messages in the queue, ApproximateNumberOfMessagesVisible = 0
At 2 minutes we are now assuming that the worker is now stuck processing its in-flight message. No action is taken yet.
At 5 minutes our ASG scale-down event triggers, and desired capacity is set to 0 (because there is no pending work)
At 10 minutes, SQS automatically re-enqueues the message due to the visibility timeout. ApproximateNumberOfMessagesVisible = 1
Shortly after, our ASG scale-up event triggers, and desired capacity is set to 1 (because there is pending work)

The 3-minute and 5-minute gaps are somewhat arbitrary, they merely highlight the need for a brief waiting period for such a fickle system.

Reference:

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-available-cloudwatch-metrics.html

HOWEVER...

The best worker is one that exits its process instead of getting stuck. That would allow for process monitoring on EC2, or for containerization. Were your worker an ECS service, it could scale up when there is work, recover from failures, and scale down afterwards. Refactoring your worker might be less effort than ongoing manual intervention for my hacky solution, and save you money!


AWS: Using CloudFront as CDN for non-AWS hosted content

CloudFront doesn't care where your origin server is hosted or even who you use for DNS.  As long as the Origin Domain Name you provide is resolvable from the Internet (and has valid SSL configuration, if enabled), CloudFront will fetch the content from that origin the same as it will fetch it from anywhere inside AWS.


How long is it appropriate to keep old feature branches in git?

Feature branches should stay until the feature is done, once done the branch must be closed/deleted. Establishing this kind of behavior should be a team goal and usually takes some time, it's just easier to forget about them in the hurry up I'm ready with my pull request mode.
Keeping all of them makes maintainability harder. You might think that each branch follows a naming convention with a map to a JIRA ticket or whatever kind of mapping you have and it's really straight forward to distinguish them, but that's not the case once you have a dozen of them and even more are constantly piling up. I highly doubt performance has anything to do, it's more like a confusion cost switch.
Regarding the commands, I do not think so.


Ansible: What's the difference between task, role, play and playbook?

No, each has a different role and way of working.
Starting with the easy ones:
A playbook is simply a list of plays.  The highest level of a playbook YAML is a list, and on that list only two things are accepted: a play definition or the keyword import_playbook, which imports a list of plays from another playbook file, as if they were defined in that place of the calling file.
The playbook is also the only thing that can be called directly from ansible-playbook.
A task is, paraphrasing the manual, 'nothing more than a call to an ansible module'.  That call definition, however, knows nothing of on which hosts it is supposed to run on.
The play is the element that ties tasks to the servers where they'll run.  The key element here is the mandatory keyword hosts.  This is the part of Ansible that tells which hosts are being affected and how.
So, the structure and behavior is as follows:

Playbook: the highest level, just a list of plays

Play: ties tasks to host lists

Tasks: definition of a call to a module
Besides tasks, a play may have pre-tasks, post-tasks and handlers, which are all task-like, and roles.

A role is a different thing, as it is not defined within a playbook.  Instead, they each have their own subdirectory under the directory roles.  The objective of roles is to organize things.  So, under the role's subdir, you can find tasks, handlers, files and templates, variables and defaults.  All of these related to that specific role.
So, roles are an easy way to encapsulate and share Ansible information.  You could have a subdir in templates per task, for example.  But if you wanted to share that task, you'd have to pick the template as well.  With roles, you can just zip the role dir and (almost) everything should be there.
Like a play, the role defines tasks and handlers.  However, roles do not define on which hosts the role will be run.  So roles must be referenced to from a play.
Roles can also declare dependencies (that is, references to other roles that need to be run before the declaring one).

PS1. Most of this information is available piecemeal on SO (except for playbooks, I couldn't find a proper definition of that on the documentation or elsewhere in the Internet), but I haven't found a comparison between them all, and it took me some time to figure this all out, so I'm sharing that by answering my own question.
PS2. I really don't like the fact that roles are named 'roles'.  It sounds a bit counter intuitive to me.  The word 'role' alone alludes to user roles or server roles, to me.  So something more related to Tower (users) or inventory (servers).  I think the rationale for that naming is that they define 'one of the roles that a server functions'.  Same could be said of plays, but roles are more abstract as they're not tied to specific hosts.  Still, I find that naming confusing.
Edit
A handler functions for the most part just like a task.  However, it is not executed by default, with the other tasks, during the playbook run.  Instead, each individual handler will be executed only if notified by a task.
On a task definition, the keyword notify identifies the handlers the task wants to notify.  If the task is executed and reports a changed state (not ok nor failed), then the handler will be marked for execution.
After all roles and tasks have run, each notified handler will be executed, in the order they appear in the file.  Check the docs for details.
Tasks will always run (unless skipped with when), and are responsible for the actual changes you want to implement.  Handlers will be run only in response to tasks making changes, and are generally responsible for finalization steps (restart or reload a service that has had its configuration changed to make it effective, for example).
You could probably replicate handlers behavior with tasks, but it is such a recurring pattern that Ansible provides handlers as a helper.


What are the levels of monitoring and their importance?

In our environment we use Appdynamics and logz.io for all levels of monitoring and it's a best tool,

Basically it monitor it for all levels,

Databse
Application especially works well on Java based services
Servers, containers
remote services
Business transactions
service endpoints, etc.,

Also it monitors all the components like, CPU, Memory, Load average, Network errors per minute, Network utilization, top 10 process usage of CPU and Memory.

All monitoring you can get it in the graph format.


Using Artifactory plugin in Jenkins declarative pipeline

You can have conditionals in your declarative pipeline by using the when-block inside a stage. There is a plugin called "environment injector" which lets you set variables outside of the pipeline-script which is nice. Also if you put the step below the other steps, it won't execute if they fail.

 when {
    environment name: 'pushArtifact', value: 'true'
  }
  steps{
     //push artifact
  }


Prometheus Alertamanger - how to silence all alerts for a given period during a maintenance?

What about Name = "prometheus", Value = ".*". That is something that worked for me.
An alternative can be Name = "severity", Value = ".*" also.


What is the behavior if docker compose fails?

Well it depends on whether the services (containers) within the docker-compose.yml depends on each other. If for example, service B depends on service A, and A fails to start up, so will B.

It can also be the case that a service logically depends on another service (without actually declaring it in your docker-compose.yml), and it not starting up causes a chain of failures. For example, say you have a web service and a db service. If the web service attempts to connect to the db service but the db service couldn't start up (config broken for example), then the web service will probably throw an exception and fail.

But no, just because one service fails, doesn't mean all other services will just be stopped. You can have two independent services, where one fails and stops, while the other is still running.


How to stop a container dying at the end of an execution?

Option --rm tells Docker explicitly to remove the container after the (main) command finished. Just leave it out.

Note that none of the examples contain this option; they only use it for -help.

As an alternative (which lets you keep --rm), you can tell Docker to mount a volume at /root. Volumes persist after the containers mounting them go away, so you can access the data written there. However, for your purposes here, a simple bind mount is quite enough; add

-v .:/root

to have it write the output file to the current (host) directory. You may have to adjust this if /root is used heavily; you probably want to use a previously empty directory on both sides.


Validating certificates with get_url or yum on CentOS 7 using Ansible

While it's true that the version of python reported there doesn't have SNI support, I thought that RedHat backported it. But anyways, if not, you can do that yourself:

yum -y install gcc python-devel libffi-devel openssl-devel
pip install pyopenssl ndg-httpsclient pyasn1

(This works for requests at least, I'm not sure about Ansible.)

On a broader note:

  But CentOS 7 ships with python 2.7.5. You can't upgrade it because get_url has the same issue with SSL and so you would have to download the python source with validate_certs=no.

That's only the case if the server you're downloading Python from relies on SNI. I imagine most of their mirrors do not.

But really, you shouldn't be compiling Python on every individual server. You should do it once, either on a machine that you freeze into an image that you launch further machines off of, or you package it as an rpm and the machines just download that and install it. This allows you to control the download process.

Even simpler: download the package on your desktop (via verified https). Copy it to your file storage. Download from there onto your servers. There's no need to download directly from python.org every time, and it's more reliable to have a local copy anyways.


EKS node cannot pull docker image from ECR: "no basic auth credentials"

Do your IAM roles that are attached to EC2 instances that are in EKS cluster have ECR iam policies?
If not please update IAM roles
Ref Link:
https://docs.aws.amazon.com/AmazonECR/latest/userguide/ECR_on_EKS.html#:~:targetText=The%20Amazon%20EKS%20worker%20node,policy%20permissions%20for%20Amazon%20ECR.&targetText=When%20referencing%20an%20image%20from,tag%20naming%20for%20the%20image.


How can I 'aws s3 sync' two buckets, which are located in different accounts

Attach a policy to the source bucket

Get the AWS account ID number of the destination account.
From the source account, attach a policy to the source bucket that allows the destination account to get objects, similar to the following:

Important: For the value of Principal, be sure to enter the AWS account ID number of the destination account.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "DelegateS3Access",
            "Effect": "Allow",
            "Principal": {"AWS": "222222222222"},
            "Action": ["s3:ListBucket","s3:GetObject"],
            "Resource": [
                "arn:aws:s3:::sourcebucket/*",
                "arn:aws:s3:::sourcebucket"
            ]
        }
    ]
}

Attach a policy to a user or group in the destination account

Attach a policy to the destination account's IAM user or group that allows the user to copy objects from the source bucket to the destination bucket. The policy can be similar to the following example:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::sourcebucket",
                "arn:aws:s3:::sourcebucket/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": [
                "arn:aws:s3:::destinationbucket",
                "arn:aws:s3:::destinationbucket/*"
            ]
        }
    ]
}

Copy objects from the source bucket to the destination bucket

After you set up the policies on the source bucket and the destination account, the destination account can copy objects from the source bucket to the destination bucket. Then, the destination account owns the objects copied into the destination bucket.

To synchronize all content from the source bucket to the destination bucket, you can run this command:

aws s3 sync s3://sourcebucket s3://destinationbucket

for more details check this link OR this link


Recommendations about release convention or standard for Python scripts?

One could consider to package the scripts. Depending on the distribution, e.g. ubuntu, windows, centos one could create a ppa, nuget or rpm respectively. Once a newer version is installed, the package manager will remove the previous version automatically.

As yum or zypper could be used to install packages on OpenSuse one could consider to create an rpm by using either spec files or using fpm.


Do I need header files once I install the php extention on Dockerfile?

You shouldn't need to statically link things to remove the *-dev packages and have everything still work. If you do, then that's a bug in the base package and the -dev package and should be reported upstream. So yes, you should in theory be able to, after installing your extensions and such, do the following and shrink the image significantly:

apt-get install -y libghc-postgresql-simple && apt-get remove libghc-postgresql-simple-dev

There are two other things you can do to improve your image sizes:

Use the --no-install-recommends switch for all of your apt-get install invocations, and explicitly list every package you depend on. Recommended dependencies are a type of optional dependency in Debian packages that are installed by default. They can often take up a lot of space, but you may not actually need most (or any) of them, so you can save space by not auto-installing them and only pulling in what you actually need. This can easily cut down image sizes multiple hundreds of MB.
Use the --purge option whenever removing packages. This will ensure that they leave nothing behind, instead of potentially leaving some config files and other similar stuff, which can also help cut down on your space usage.


Trying to migrate CI/CD process from Jenkins to Gitlab Runner + Docker on Windows

have anyone tried this scenario (gitlab runner + docker on Windows) before, and have some feedback if it works or not?

I installed and registered a GitLab Runner on my Windows 10 laptop following the instructions from GitLab Install GitLab Runner on Windows

At first, I had trouble pulling the Windows container image:

PS C:\GitLab-Runner> docker pull mcr.microsoft.com/windows/servercore
Using default tag: latest
Error response from daemon: manifest for mcr.microsoft.com/windows/servercore:latest not found: manifest unknown: manifest unknown
PS C:\GitLab-Runner> docker pull mcr.microsoft.com/windows/servercore:1809
1809: Pulling from windows/servercore
no matching manifest for linux/amd64 in the manifest list entries
PS C:\GitLab-Runner>

After selecting the "Switch to Windows containers" option in my Windows Docker (in the top-level menu that opens from the Docker icon in the toolbar), I was able to docker pull the Windows servercore image.

However, I was unable to run GitLab Runner on my fully patched and updated Windows laptop:

 ERROR: Preparation failed: unsupported Windows version: Windows 10 Pro Version 1909 ...

It's too new!  As I read GitLab's Windows version support policy, GitLab should support 1909 for 18 months; but obviously that is not the case yet. 1903 is the most recent supported version.

However, as of January 2020, GitLab has Windows Shared Runners (beta)

I tried that and it works. Example .gitlab-ci.yml:

my_job:
  script: systeminfo
  tags:
    - windows

  Is my plan of decreasing Jenkins usage and using more Gitlab runner (for this project at least) feasible?

Sounds good to  me.

  Do you have a better approach on how can I improve to a scalable Windows project CI/CD process, other than using gitlab runner + docker?

No.  I think you're on the right path.


Jenkins Dockerfile declarative pipelines: getting the built image tag

As I understand you want to add a variable to the docker image tag,

in this case, you have to get an environment variable from Jenkins and use it in your image label.

    pipeline {
  environment {
    registry = "docker_hub_account/repository_name"
    registryCredential = 'dockerhub'
  }
  agent any
  stages {
    stage('Building image') {
      steps{
        script {
          docker.build registry + ":$BUILD_NUMBER"
        }
      }
    }
  }
}

pleae use this URL for more info


I would suggest to only run stateless apps in k8s. One should run statefull apps like databases in dedicated database solutions like AWS RDS.


Is there a conceptual limitation to have a clone stage function in Bamboo?

Stages group jobs inside plans. So technically they are cloned when the respective plans are cloned.

Which means that missing the ability to clone a specific stage should be just a UX/usability/functionality issue, not a conceptual one.

There might be a non-trivial implementation details when cloning stages - they may carry inside stage sequencing information, as in "pointers" to the stage objects preceeding and/or following them inside the plan "pipeline", which might be a bit more difficult to handle when cloning just one stage in the sequence. These would be easier to handle when cloning entire plans as the entire sequence of stages is cloned, there will always be a 1:1 relationship between the stages.


How to remove a non-existing GitHub Ansible role from Galaxy?

You have to pass only the repository not the full HTTP URL of the repository when you want to delete it from what I've read in the documentation. So the command should be:

ansible-galaxy delete 030 ansible-firewall


Jenkins pipeline fails on script approval, although it was approved

The issue was with the wrong UID and GID of the user. after I've changed it from jenkins to 1000 1000 recursively, issue solved.


What OS should a master node CI virtual machine be?

I think the major deciding factor is the expertise of you and the rest of your company in the chosen OS. If you are a Windows shop, and your company is willing to leverage the cost of the node, it's probably best choice to host it on a Windows VM. For choosing a Linux distro, I would see what is most common across your company. In my opinion, it is not worth the benefit of creating a vm with a one off distro just because of a few particular features, unless those features of the distro are absolutely necessary.


What are further known DevOps operation models?

Your question doesn't seem to make any assumption about the platform/OS it is about. Which is why it may make sense to add an answer about how part of such toolchain looks like in a mainframe environment. My answer is based on using the SCM product where I'm most familiar with (not sure if it's needed to disclose the product name).

Your example about "Plan, Code, Build, Test, Release, Deploy, Operate, Monitor" seems to be missing these crucial phases:

Enforce predictable problems in production to be addressed before they actually have an impact in production (sounds like sales talk, no?). You might think this is just "impact analysis", but that is not it:

Impact analysis is what you do in the plan-phase (we all know that, right?).
Audit (= name of the phase we use for it), is around the time that the developer thinks/claims to be ready. Something like "unless there is anything else that may have happened in production while I was working on my change, I'm done and recommend to move forward". Think of things like "version regression issues". Eg because last night a fix was applied to production and you didn't retrofit that fix, so if you move forward with your change, you'll wipe out the fix again ... bad luck if you do!

Give me a red button, so that if anything else fails (and production is impacted), I just have to push a magic button, be patient (for a few secs) and be 100% sure a rollback completed with 0 rebuilds of any kind (= Backout is the name of the phase we use for it).

This boils down to "component level backups" of any files/components you will be updating by your change in production, whereas these backups are made at the very moment the change is going to be applied. and this for any type of artifacts included in the change.

If later on (5 mins, 2 days or 1 week later) something goes wrong in production, and it turns out to be caused by your change, all you have to do is to have an automated procedure (the red button!) in place that you can launch, and which simply restores the backup you should have available.

Attention, if there were database updates involved, this backout may not be an option anymore. While things like "backups overlaid" (because of concurrent changes/fixes got applied to production) is another scenario where the red button is no longer available.

Easy, no?

PS: I leave it to everybody's own judgement if my answer is yes or no DevOps-compliant.


How to pass groovy output to pipeline variable

You can send the output of your python script to stdout and capture it in a environment variable and that variable can be passed to other stages in the pipeline. Also, the environment variables can be used inside the scripts.

You can do something like this:

def call(Map parameters)
{
    def CREDENITAL_ID = parameters.secret
    def DOMAIN_DESCRIPTOR_URL = parameters.url
    env.artifacts = sh(script: "python amd_distribution_input_transformation.py", returnStdout: true )

    }
}

Here, the output of the script amd_distribution_input_transformation.py will be stored in the variable artifacts and will be available further.


What are pros and cons of Hackathons on implementing a DevOps culture?

If the project is for a startup's minimum viable product or some other sort of sanity-check/proof of concept effort - hackathon before DevOps is often recommended - basically in the "fail-fast/fail cheap" lean startup strategy.

Some might recommend hackathon before DevOps in other situations as well, for time-to-market reasons. But if DevOps doesn't followup fast enough that initial market bridgehead can easily collapse. Possible reasons for that include:

that early momentum can be ephemeral - easily lost as soon as the DevOps process starts to be established ("it's too hard, it's slowing us down")
typically hackathons cause accumulation of technical debt. The cost of managing and eliminating it can outweighs the benefits of that early momentum.


How to run database migrations in an auto-scaled group

How are you doing your deploys?  Nuke + rebuild the ASG (whether one node at a time, or by replacing the whole group at once), or do you have a script that redeploys all of your active nodes?

And also, how are you triggering your deploys?

Ideally you want to be running/triggering deploys from some form of CI server like Bamboo or Jenkins.  If you're doing this, you can run your DB migrations from the build server (or its component build agent instances).

This would allow you greater control over your environment, as you could, for example, restore DB from snapshot if the deployment fails.


Any tools available for rollout planning / checklist

Document the functional process

When I create a new project and want to deploy it I always start to right down the manual steps in a README, but one could also create a txt file or sheet like you did.

Feel the pain

At a certain moment when I have deployed the project manually the pain of certain steps will become clear. For example, I add per step how much time it takes.

Automate the high ROI steps

If for example a certain step takes half an hour an one deploys twice a week, one could investigate how long it will take if one automates this step (investment) and what time it will save (return). One could decide to create issue tickets and automate steps every sprint.

Example deployment plan

For one of my current projects I have to deploy multiple microservices. I will not provide all details, but initially the plan was two pages long. Now I only use a Continuous Integration/Deployment (CI/CD) plan and a lot of abstractions. If the deployment will fail some day I could still read the initial plan so that the functional flow is still clear.

Configuration as Code

Another benefit of automating the manual step is that one could save the code in a Git repository as a pipeline script. When another person reads the scripts then it becomes clear how the project is deployed and one could also trace who made some change, what modifications were done and when.


How to create a centralised dashboard to monitor multiple cloud (Amazon and Google)?

I think what you are looking for is an Open Source project that can take inputs from both Amazon CloudWatch and Google StackDriver; there isn't a huge amount out there at the moment, but I will detail what I know.

I have made the assumption that you know how to import your application telemetry into the solutions below.

Open Source

Open source solutions to this problem have been lagging behind as late due to favourable prices of SaaS and built-in solutions.  The products below are constantly evolving so if they don't do what you need now they may be in the future.

Elastic Stack, formerly known as the ELK stack is made up of three individual products from Elastic.co: ElasticSearch, LogStash and Kibana.  It is possible to ingest data from CloudWatch with a plugin and to export StackDriver data to Google Pub/Sub and then ingest into LogStash.
Grafana, is slightly better than Kibana at displaying time-series data. However, dosn't have as robust an ecosystem around it.  Grafana has built in support for Amazon CloudWatch, Google StackDriver is not supported as yet. However, someone is working on it in the form of a plugin.

Software as a Service (SaaS)

Software as a Service monitoring solutions is a multi-billion dollar market, as large enterprises will readily pay for a SaaS or off-the-shelf solution to their dashboarding challenge.

New Relic, a very mature product with extensive cloud support, including ingesting of CloudWatch metrics and Google Cloud Storage Metrics.
Splunk, a bit more focused on logs than metrics, however, has good support for AWS CloudWatch and Google Cloud Platform.
Google Stackdriver, obviously has good support for Google Cloud Platform, also it is capable of reading performance metrics in from AWS.
DataDog is my personal favourite, it has a vast range of integrations including support for Amazon Web Services and Google Cloud Platform.

Single Pane of Glass

In the last three years, various products have gone to market providing a "Single Pane of Glass".  The advantages of these products are:

Heterogeneous Platforms - you can use many different monitoring tools and products and then unify them with a Single Pane of Glass.
Data Normalization - most monitoring tools have idiosyncrasies in the way they present data a Single Pane of Glass helps to normalise these so that metrics from Google, Amazon and your App make sense when displayed in the same graph.
Business View - a Single Pane of Glass usually has the capability to integrate with non-technical data sources.  As an example displaying data from financial tools to show a measure of business success.  This view makes for some compelling C-Level dashboards that demonstrate how technology is providing value to the firm.


How to test VM provisioning script without provisioning

I can see some options:

Use Vagrant to create your VMs; it separates the process of creating the VM (including the base OS) and the actual provisioning. It also has some options to run certain provisioning steps at certain circumstances only.
Use Ansible, Puppet or something like that to switch to a provisioning mode where you do not do the same stuff everytime, but only what is needed. This means that you can start the job, and at the first failed part, stop. Fix that part, then continue.
Use Docker. This is slightly different than the Vagrant/Ansible approach in that it creates containers (which you don't actually need, as far as I can tell). It has the benefit, beyond the Ansible approach, that it gives you a very fine grained step-by-step development process. I.e., if one step fails, you still have all the images leading up to that, so during development, with a bit of discipline, you become very, very fast indeed.

All of these tools do much more than what you need, but all of them give you a way to do your job incrementally. Vagrant, Ansible and Docker are pretty easy to learn, as far as I'm concerned (as long as you are in Dev/Test mode, the "interesting" parts start when you go to production). Ansible is very minimalistic and needs nothing except a ssh connection. Vagrant and Docker might not be feasible in your infrastructure, you will quickly see.


Jenkins Slaves cannot connect to the Master Jenkins

It depends on how you connect you slaves via ssh or jnlp,
in any case: check if you are able to connect that slave and try telnet it.


How to use kubernetes_service data provider in Terraform?

The provider documentation you quoted here seems to have a mistake. It talks about using the two providers together in the same module but it seems like it really means in the same configuration.
The problem arises because Terraform needs to configure a provider in order to plan with it, and so everything inside a provider "kubernetes" block must be known at the time of planning. This means you can't use anything in there which isn't created yet.
That documentation is trying to tell you that you'll need to split the configuration that declares your Kubernetes cluster from all of your configurations that configure or make use of the Kubernetes cluster, and then you can terraform apply the creation of the cluster first, and then separately terraform apply your other configurations that interact with it using the kubernetes provider.


It depends on several factors.

If your images can be public or must be private. If you can make a public image, docker hub could work.

With ECR you will have a fine grain control of your permissions, integrated with IAM

On the other hand, ECR it's only available on one region so if you pull an image from a different region you will be charged for that traffic and the latency could be a problem so if you need a multi-region registry yo should implement that mechanism by yourself.

In brief, I wouldn't use public registry for production applications. You'll have more control with a private registry like ECR or other 3rd party solutions.


Is web delivery for Node.js apps a thing?

I'm not aware of a project that provides that out of the box. For local development, it is common to have hot reload. If something at the file level changes, it restarts the server. That is somewhat related, as you could trigger the updates on the file system remotely. Installing libraries could also be simulated by directly updating the  node_modules folder.

Beside that NodeJs itself provides ways to run arbitrary code without process restarts. One way is to use eval (as part of JavaScript). The other is to reload a file. Normally, require caches the file after it executed it, but you can purge this cache (require.cache), and require the file again. That is what the decache library does.

In theory, NodeJs provides all the tools to allow remote code being applied. Of course, from a security perspective, you open yourself to all kind of attacks. Make sure requests come from a trusted source.

  But for web-delivery, I think a workflow for deploying a node.js-centric application and keeping it up to date would use some other mechanism. What is that mechanism? If docker is the way, then maybe I just need to refocus my question to find out how to auto-update docker containers

I would also see it less as a matter of updating a NodeJs application (driven by the application itself), but rather as  of a good deployment pipeline (driven from outside). In the end, deployment is nothing else than pushing changes to production.

There are a variety of deployment approaches for NodeJs applications:

Classic deployment on a server
Docker based deployments (e.g., Kubernetes)
Serverless (e.g., AWS Lambda, Serverless Framework)
Platform as a service (e.g., Heroku)

If you are not sure, the easiest to get started with are serverless deployments (supported on all major providers: AWS Lambdas, Google Cloud Functions, Azure Functions), or platform as a service solutions (e.g., Heroku). Kubernetes (as a popular framework for deploying Docker containers) is powerful, but has a steep learning curve.


What are tools that lend themselves for backup and recovery in a CI/CD environment?

The things that would be backed up would be the config files of the servers, the database and uploaded content as these will almost always be unique and necessary. The containers, VMs, even the webapps would NOT be backed up as these should all be uniform and constantly updated under the CI/CD principles. Currently I use a combination of scp and tar into an archive directory sorted by date and machine. I like to find something better if possible.

Config files, containers and VMs should be backed up and versioned by a tool like git or SVN if you are doing configuration management properly.. Backing up config files and versioning them is the whole point of configuration management tools like, salt, puppet, ansible, and chef.

As you duly note however, dynamic content such as database data and similarly dynamic data is a more challenging issue. These however, are outside of the scope of configuration management, CI, and CD principles. These principles purposefully remain silent on dealing with these issues: because they have no new answers for these problems. Instead, you should use the many battle-tested techniques and strategies for dealing with this kind of data instead of expecting CI/CD/CM tools to solve problems that they were never intended to address.


Jenkins won't substitute variables when trying to check out from perforce

Your string isn't being interpolated at the correct time because it's in single quotes.  Instead it's being passed to the checkout method as an un-interpolated string.  By the time the checkout method interpolates the string, the perf variable is out of scope and so interpretation of this variable falls back to a blank string.  Something like this is what you (probably) want instead:

view: "\"//depot/Software/${perf[i].ID}/...\"" + '//jenkins-${NODE_NAME}-${JOB_NAME}-${EXECUTOR_NUMBER}/...'

Note how this concatenates two strings, a double-quoted string and a single-quoted string.  This ensures that each string gets interpolated at different times.


How to change Xmx settings for sonar runner?

You can simply put a sonar.properties file under /opt/docker-sonar/conf/. This file will be available inside the container under /opt/sonarqube/conf/, because the folder gets mounted as volume.
A full example for a sonar.properties file can be found on github. However all you need to enter is:
sonar.ce.javaOpts=-Xmx<XMX_VALUE -Xms<XMS_VALUE> -XX:+HeapDumpOnOutOfMemoryError


What is the optimum number of people in a weekly oncall schedule?

well, this is not your answer but I hope it helps ...
I lived both situations when I was single and no kids.
When the staff was short my boss tried to motivate us by talking about the extra money
and giving half day off. (I liked)
When the staff of complete I was getting on-call once a month and the money wasn't good enough to pay for the trouble. (I hated it)
I recommend to assess where your team is, personally and professionally, and put a few suggestion to be voted by them. Then give them the ability to self manage their schedule.


availability calculation of a azure service fabric stateful application

Virtual Machine Scale Sets (VMSSs) don't have a SLA of their own:

  Virtual Machine Scale Sets is a free service, therefore, it does not have a financially backed SLA itself. However, if the Virtual Machine Scale Sets includes Virtual Machines in at least 2 Fault Domains, the availability of the underlying Virtual Machines SLA for two or more instances applies. If the scale set contains a single Virtual Machine, the availability for a Single Instance Virtual Machine applies. See the Virtual Machines SLA for more details.

The above statement points to the fact that VMSSs are just an orchestration layer over the top of Virtual Machines and as you quite rightly pointed out results in an availability for a VMSS of 99.95%.

In terms of the storage calculation, there are several parameters here:

Whether you are using Hot or Cold storage, the availability for Hot storage is 10 times that of cold storage.
Whether your application is read or write heavy, what you need to consider here is that there are separate SLAs for reading and writing.

Let's make some assumptions:

Storage is, in general, cheep.  Therefore, only ever use the Hot Storage Tier.
Managed Disks are, in general, easier to use.  Therefore, you are limited to only using Locally Redundant Storage.

With the above two assumptions your storage SLA is 99.9%.

Multiplying these two figures together you get, 99.985%, this is still kind of low, but higher than your 98% figure above.  You can increase this number by hosting Service Fabric across multiple regions:

This would give you system availability close to four nines, i.e. 99.9898%


Ansible Playbook fails start Apache Cassandra

I found my fault.
It's necessary define a cassandra.service to Systemd manages Cassandra:
path: /etc/systemd/system/cassandra.service
And adjust like you installation
[Unit]
Description=Cassandra Cluster Node Daemon

[Service]
Type=forking
User=cassandra
ExecStartPre=/usr/bin/echo "Starting Cassandra Daemon"
ExecStart=/usr/sbin/cassandra
ExecStartPost=/usr/bin/echo "Cassandra Daemon Running"

ExecStopPost=/usr/bin/rm -rf /cassandra/saved_caches;/usr/bin/echo "Cassandra Daemon Stopped"

SuccessExitStatus=143

[Install]
WantedBy=default.target


Running unmanaged Hardware Security Modules (HSMs) in the Cloud

So having gone backwards and forward over this for a couple of weeks, Azure has confirmed to me in-person that the only way to utilise FIPS-140 Level 2 certified hardware security modules in Microsoft Azure is to use Azure Key Vault.


Can Ansible alone be a complete CI-CD tool?

This is totally possible.  Of course you can achieve it without jenkins or similar tools being required.  We're always free to reinvent the wheel.  The question becomes is it worth the effort?  When it helps you avoid jenkins I'd be inclined to put the effort in.  (I'd also suggest looking at Concourse before doing jenkins again, but that's not what this question is about...)

For your case you need two things:

a playbook that automates your build using ansible.
a git hook to invoke your ansible playbook, probably post-commit.

If you are using gitlab or github you should look at their specific documentation for creating hooks, but otherwise the generic help I linked to should do it.

To answer your specific questions:

Yes, use the git hook.
Sure.  You probably want ansible to do a fresh git checkout first.  Eventually you will want to add any prerequisites to be verified too.
You don't have to assume anything, just add these steps to the end of your playbook.  If the build fails, it will stop before wasting effort on testing anything.  You could split build, deploy, and validate into their own roles within one larger playbook.  This will help make each of the sets of tasks easier to manage and you could call them manually if needed for debugging or emergencies.
You could do this in a myriad of ways.  I'd create a text file with a date/time stamp and the results inside.  You could write it to a database or K/V store.
I'm not sure what you mean.


Is it possible to give only view permission for Jenkins monitoring-plugin

No :). please see Jenkins issue 14750:
https://issues.jenkins-ci.org/browse/JENKINS-14750


Necessity of mirroring git repos for small team considering switching to Azure

As you have realised Git is decentralised by design.

Using LAN repos

If Azure DevOps goes down, or your office internet connection, you can still use LAN repos. These could be simple file shares. You can also use emails to share commits; nothing I've tried but Git has native support for it). I have however pushed and pulled commits directly from and to my colleagues computers using file shares (because our online Git provider back then was extremely bad and slow (and the repos were many GB).

If you don't have automatic sync (which you can setup using pipelines/CI) between Azure DevOps and GitLab then the most up-to-date repos will reside on the developers computers which makes it easy to push the repo to a new location when needed (simply add the new remote and click push). However, if some repo hasn't been used in a long time it'll probably not exist on any developers computer and you'll therefor not be able to access it.

If you want to be disaster-ready I'd suggest hosting an Azure DevOps Server instance on your LAN that's synced with the cloud. Azure DevOps Server is capable of processing the very same yaml pipelines as the online version. You'd be protected against much more than just Azure DevOps Services outages.

The maintenance costs of having Azure DevOps Server as a backup for a small team shouldn't be much different than having GitLab as backup.

I'm not recommending these solutions, rather I'm just discussing them.

You might want to have a look at the Azure DevOps Proxy Server.


How to handle item not in list in ansible variable?

use the condition

- when: variable is defined
  run_module_when_defined:

or

- name:
  block:
    - task1
    - task2
  when: variable is defined


How to schedule a Jenkins Job dynamically based on the time value in database

Jenkins itself isn't very good at using external data sources to drive dynamic job scheduling.  The reason for this is that job configuration, including scheduling, is built statically, either manually via the web UI, or with code via a Pipeline script which is run at job execution time.  Even if the Pipeline script builds the job configuration in a dynamic way, the configuration itself will be locked in until the next job execution.

The easiest way to have flexible build schedules is use an external source to drive your builds.  The hook mentioned in the comments would be ideal, but it sounds like that is not possible for you.  Another idea would be to have a cron job on your database server that runs once a day, grabs the build time from the database, and schedules the build for that time.  I'm sure this isn't a great solution, but it's just an example.


I ended up following this path
Rather than use cron, i used a scheduling framework (in my case Bree.js) which allowed me to "jobify" the tasks and also execute these tasks in separate processes - this allowed me the vertical expansion needed (ie: using more processor cores)
A docker container contains the dispatch (scheduling) and jobs - so once the container is started, jobs (and associated processes) are created/managed within that container
Eventually, when I need to scale horizontally, the "dispatch" function will morph to running a job whose sole purpose is to post an event  (eg: Azure service bus).  I will have an orchestrator running various containers who read from the queue and process accordingly.


Good IDE or plugins for typical DevOps functions?

I personally really like PyCharm (same guys who made IntelliJ and Android Studio).

I mostly use Ansible, Bash, and Python.  Has good plugins + syntax highlighting for Bash (doesn't everything at this point?), and Ansible.  Since it's a native Python IDE, it's also very convenient when writing or troubleshooting python scripts, especially since you don't even need to exit the IDE to run them.

It also has integrations through plugins:

Chef: https://plugins.jetbrains.com/plugin/7548-chef-integration

Ruby: https://plugins.jetbrains.com/plugin/1293-ruby

Both should in theory allow full IDE-like experience with both languages (i.e. resource completion, syntax highlighting, jumping to references, etc).

Really, you can't go wrong with anything by JetBrains if you choose to use any of their other IDEs like IntelliJ.


create entry-point script to get all env variables starting with VUE_APP_ and take it's value

While it's not clear what your trying to do (the script you have so far for the bit you are having trouble with would probably be handy), I think you are essentially just trying to parse your environment and manipulate the variables at the start of your script?

I don't see why add something like this to the first line of your script wouldn't work:

eval $(typeset -p | grep -E 'declare -x VUE_APP_' | sed -e 's#declare -x VUE_APP_#export VUE_APP_#g')

But then I also don't know what shell you're running or anything else about the container, so I'm not sure..


How to deploy multiple dependent micro services

Such question could be an indication of a poor architectural slicing into microservices. From What are Microservices?:

  These services are built around business capabilities and
  independently deployable by fully automated deployment machinery.

The key point missed in such case would be their independently deployable aspect.

The point could also be simply misunderstood: it doesn't necessarily mean that each microservice should be capable to perform its entire functionality by itself, it just means that it's able to gracefully bail when some other microservice its functionality depends on isn't available, maybe keeping track of "todo" stuff for later execution, when that microservice becomes available.

Seen from a different perspective: handling unavailability of external service dependencies should actually be part of each microservice functionality - so one could consider the microservice fully functional even when its runtime dependencies aren't satisfied :)

It's worth noting, I think, that in the above comments I'm referring to microservices unavailability regardless of its reason, be it deployment dependency order or something else (outages, for example).

Another thing to consider would be the finer difference between deploying the microservices and bringing them into action by switching the traffic to them. Switching traffic is often faster than the deployment itself, so the availability at the entire system level can be increased by first deploying all microservices and only after that performing the traffic switch for all of them.


Docker Node.js Error: Cannot find module

volumes:
      - /usr/app/node_modules

This volume could be overwriting what you have installed during the RUN npm install stage of the Dockerfile. As far as I can see you're not doing any installation when starting the container, so the contents of the mount are empty. If you remove that mount, the node_modules path will be populated with what was installed during the docker build (as defined by the Dockerfile) phase.


How do I create a Docker container which just waits to be shelled into?

Disclaimer: This is very much an anti-pattern since it looks like you are trying to create something similar to a VM, or at the very least a pet that you modify in place rather than making all your changes via code and redeploying a new container for each change.

That disclaimer aside, you just need a command that will hang indefinitely. The most common one I've seen in a tail -f /dev/null, e.g.:

FROM ubuntu:18.04

RUN apt update \
 && apt install -y \
      cmake \
      curl \
      git \
      jq \
      libasound2-dev \
      libglu1-mesa-dev \
      libgtk2.0-dev \
      libjack-jackd2-dev \
      libx11-dev \
      libxcursor-dev \
      libxinerama-dev \
      libxi-dev \
      libxrandr-dev \
      zlib1g-dev

CMD [ "tail", "-f", "/dev/null" ]

I've also rearranged the Dockerfile to merge the update and install commands, this is a best practice for Dockerfile's since it avoids having a stale apt update command used from the cache when you change the apt install command months later. Also, putting each package on a separate line is better for version control to see what changed if you add a single new package.

When using this, you would docker run -d --name your_container your_image to run it in the background, and then docker exec -it your_container /bin/bash to open a shell.


How to check Jenkins build executor free memory?

Have a look at the Monitoring plugin. Here are some more details about it (from the linked page):

  Charts of memory, cpu, system load average, http response times by day, week, month, year or custom period
  Statistics of http requests with mean response times, mean cpu times, mean response size by request and by day, week, month, year or custom period
  Errors and logs
  Current http requests
  Threads
  Heap histogram (instances and sizes by class)
  Http sessions
  Process list of OS
  MBeans
  Actions for GC, heap dump and invalidate session(s)
  Report in html or pdf
  In English, German, French, Portuguese or Chinese
  Jenkins security
  For Jenkins nodes (slaves in general):

  The report for the nodes is available at http://yourhost/monitoring/nodes
  Charts aggregated for all nodes of memory, cpu, system load average, number of running builds, build queue length, build times by period
  Detailed statistics of the build times and of the build steps by period
  Threads, process list and MBeans for each nodes
  Heap histogram aggregated for all nodes

  For each individual node (each node in http://yourhost/computer), reports and actions are available from the "Monitoring" page in the contextual menu or in the detail of the node:

  Threads, process list, MBeans of that node only
  Heap histogram of that node
  Actions for GC, heap dump

  And more...

Credits: Assaf Lavie (who first mentioned it in a comment, but for some reason didn't post that as an actual answer ...).


Specify default arguments for Ansible playbook

That's what the configuration file is for. Create an ansible.cfg in the directory from where you run ansible, like so:

[defaults]
inventory = hosts
ask_vault_pass = True

[privilege_escalation]
become_ask_pass = True

See also http://docs.ansible.com/ansible/latest/reference_appendices/config.html

The ansible.cfg in the working directory precedes /etc/ansible/ansible.cfg. The full search order is described on the page linked above. Command line options you pass on invocation override all configuration files.


What could be a valid definition of DevOps to introduce it to a novice?

DevOps in a nutshell

From Wikipedia:

  DevOps (a clipped compound of "software DEVelopment" and "information technology OPerationS") is a term used to refer to a set of practices that emphasize the collaboration and communication of both software developers and information technology (IT) professionals while automating the process of software delivery and infrastructure changes.

  It aims at establishing a culture and environment where building, testing, and releasing software can happen rapidly, frequently, and more reliably.

From Overview:

  Venn diagram showing DevOps as the intersection of development
  (software engineering), operations and quality assurance
  (QA)

While there is no single "tool" for DevOps, but rather a set of tools, also known as a DevOps toolchain:

  Illustration showing stages in a DevOps toolchain

Illustrations of DevOps

Below are a few quotes from some of the questions on DevOps.SE, which all seem to somehow fit / confirm part of the DevOps description above:

From 'What is the difference between SRE and DevOps?':

  ... DevOps is about maintaining code ...

From (an answer to) 'How do I hire a good DevOps, fitting my company?':

  ... It is a common misbelief that “DevOps” is a role in a company. The word merely denotes an organisational shift in software companies ...

From 'Does my organization need adopt Agile Soft. Dev. before adopting DevOps?':

  ... there are still organisations which are not practicing Agile in their software development, who might be interested in adopting DevOps ...

DevOps is NOT a role

Below are a few quotes from some of the questions on DevOps.SE, which all seem to illustrate that DevOps is NOT a role:

From (an answer to) 'What is the difference between Sysadmin and DevOps Engineer?':

  ... DevOps is not a role (when used as such it's more a buzzword than a real role).

From (an answer to) 'How do I hire a good DevOps, fitting my company?':

  ... It is a common misbelief that “DevOps” is a role in a company. The word merely denotes an organisational shift in software companies ...


Update production server (apache, wordpress, mysql) on docker containers, from staging server

My thoughts based on your described scenario are:

Your tarball strategy, as described, might sound safe but is prone to risk in my opinion. A better approach is to version-control the WP files you will modify, while keeping the core intact and within the Docker container. Here's a reference I have used in the past that talks about version-control strategies for WP. You would of course need to spend some time and understand which folders/files to version-control for your specific site.
The version-controlled files would be checked out on the host machine (which runs Docker) and be volume-mounted within your Docker container.
Everytime your changes are validated in Stage they get pushed to Git. Getting these changes into Prod would essentially be a git pull.
If you wish you might also use a tool like composer as part of your deployment workflow. A dependency manager should be part of any deployment worklow.

I would also recommend tying all this together with a tool like Ansible, i.e. use Ansible to

update local git repo
run the composer bits
ensure docker container is running

Disclaimer: Although I've worked on DevOps automation for Wordpress sites, I am by no means an expert on Wordpress, so suggestions and clarifications welcome.

Hope this helps.


Do not show all docker containers to a particular user

Hi and welcome to DevOps SE!

To enable Role-Based Access Control (RBAC) for Docker Engine, you need the UCP (Universal Control Pane) available through Docker EE.

https://success.docker.com/article/rbac-example-overview

A free alternative could be also the Kubernetes module rbac.authorization.k8s.io.

Side note: To control docker commands available to a user, use AuthZPlugin for the Docker engine (there are open source implementations of this API).


Encrypting docker images (end to end); transporting over offline channels

Google Container Registry is a potential option.

Registry is private
Image layers are stored on a Google Cloud Storage bucket, which is encrypted by Google (RSA) and can be additionally encrypted with a client key (RSA)
Access control is granular down to each individual object stored on GCS bucket
Client-side authentication is enforced via oauth2. Clients configure docker to use the google cloud sdk as a credentials manager
Images are pulled via ssl connections

The fact that the registry is part of Google's Cloud Platform also means you can build up to some other interesting things. For example, you could set up a VM to serve as reverse-proxy and set up the GCP firewall to only allow 443 from a certain set of IPs (like your clients' offices).


Ansible URI module and skip SSL validation

You seem to be passing force_basic_auth: yes but no user, nor password.
Example from the Ansible docs:
- name: Create a JIRA issue
  uri:
    url: https://your.jira.example.com/rest/api/2/issue/
    method: POST
    user: your_username
    password: your_pass
    body: "{{ lookup('file','issue.json') }}"
    force_basic_auth: yes
    status_code: 201
    body_format: json

The 403 is usually Basic Auth failing. If it's not, it could be the API server you're running requiring some sort of authentication.
If the curl -k command includes something like https://user:password@dev-box.local/api/ or the --user <user:password>, it should work. Or maybe it's using data from .netrc?
I hope this helps.


What is the difference between packages and artifacts?

Mainly an artifact is the result of of a build phase, this mean a package is an artifact of a kind.

A package is usually a way to install a software or application, it includes the software itself and some intelligence to setup and configure the software.

Calling an artifact a package usually comes when the artifact (whatever it is, from a .deb to a simple zip file) contain more than just the application, it could be a simple script to help configuring the software and setting proper permissions or other softwares needed.


Three solutions to structure a deployment with Ansible?

I would recommend to structure the code as defined in the Ansible Best Practices document:

and to follow the structure that major contributors to the ansible galaxy platform use, like geerlingguy.

If one will follow a significant different structure then it will be really hard in the future to reuse roles from the ansible galaxy community and one has to reinvent the wheel.


What are feasible quality criteria for Git repositories?

Who is responsible for checking that people don't check in commits that have large files? The same people who are responsible for checking that the commits aren't bad in other ways: everyone.

If git is new in your company, make sure that the git education includes this sort of thing. After that, it's up to code reviewers to notice mistakes and correct them.

If it becomes a repeated problem, then whoever is in charge of developer tooling can write some checks that run every night or whatever and let people know that something slipped through. But that seems like overkill in most situations.


Have you heard about the distroless?
See this link too.


Docker daemon error on joining swarm node - could not find local IP address - unknown port

Could you please try the commands above between all hosts ?

INSIDE HOST A:

nc -zv HOST_B_IP_ADDRESS 2377,4789,7946
nc -zv HOST_C_IP_ADDRESS 2377,4789,7946

INSIDE HOST B:

nc -zv HOST_A_IP_ADDRESS 2377,4789,7946
nc -zv HOST_C_IP_ADDRESS 2377,4789,7946

INSIDE HOST C:

nc -zv HOST_A_IP_ADDRESS 2377,4789,7946
nc -zv HOST_B_IP_ADDRESS 2377,4789,7946

As explained in https://docs.docker.com/engine/swarm/swarm-tutorial/#open-protocols-and-ports-between-the-hosts, some ports must be available between hosts in order for Docker Swarm to work correctly.

The error reported sounds like a port related problem.
Maybe is there another application in one of these hosts already using one of the ports needed by Docker ? Maybe a firewall configuration ?


Automating this kind of behaviour is a quite risky since you have ResourceQuotas usually set up as a constraint for a namespace and from there on the pods must be aligned with it.
You might end up with a pod non being scheduled at all.


Automatic production deployment for a Node.js app on RHEL 7?

Ok, let's assume you'll pull the latest version of branch prod from a git repo, here's what it would take with chef on a basic illustrative purpose:

git "/opt/my_application" do
  source "https://<git_host>/<you_repo_url>/<repo.git>"
  revision "production"
  action :sync
  notifies :run,"execute[app_config]"
end

execute "app_config" do
  command "npm install"
  cwd "/opt/my_application"
  action :nothing
end

For more advanced use cases you can have a look at the poise application cookbooks, it includes a javascript plugin for node.js application.

chef is fully open source, either client or server side (out of the fancy UI) and on all OSes including windows.


Ansible cloudformation vs cloudformation

You wouldn't need to use Ansible for AWS cloudformation if you are fine with just using AWS cloudformation.

Ansible integrated AWS cloudformation so that you can convert your local Ansible files into an AWS cloudformation easily.

I'd suggest using Ansible since it provides a great way to test your stacks locally and then you would make the conversion to AWS cloudformation when ready. This is usage dependent however.


Jenkins node (agent) on shared web server - what are the right permissions

A not very devopsy, but quite simple and effective solution with no security implications (no credential exchange or authentication required) would be to establish a file-based "message" exchange scheme, in a "mailbox" - a well-known filesystem location (setup once and owned by root) with 2 directories:

one owned by the jenkins user which creates request files containing deployment request information, one for each site
one owned by the apache group where each site dedicated users create their own response files containing request handling information for deployment requests for their site

When jenkins processing reaches a deployment stage for a particular site it creates a corresponding request file with the necessary information in it.

Each site user periodically (cron-driven, for example) checks for request files pertaining to their site, handle the request as appropriate, following their own site policies and provide status updates in the respective response files, which the jenkins user checks periodically.

When a request handling is completed the jenkins user removes the request file, signalling that it received the "message" and then the site user periodical job can remove the corresponding response file.

The names of the request and response files can be used to encode the particular site and request identification, so that the periodical checks don't have to fumble through multiple files.

The scheme can easily work across machines (if, for example, some of the sites are migrated to other servers) simply by placing the "mailbox" on a shared filesystem accessible from all those machines.

OK, an example, as requested. Just a basic skeleton, in python, hopefully self-documenting.

Prerequisites:

sudo mkdir -p /var/message_box/requests
sudo chown jenkins /var/message_box/requests
sudo chmod go-w /var/message_box/requests
sudo mkdir /var/message_box/responses
sudo chgrp apache /var/message_box/responses
sudo chmod g+w /var/message_box/responses

The mailbox.py file:

#!/usr/bin/python2.7 -u

import logging, os, re, getpass, sys, time, yaml

class Mailbox(object):

    base_dir = '/var/message_box'
    request_filename_format = '%s.%s.yaml'  # username.id.yaml

    def __init__(self):
        pass

    @property
    def request_dir(self):
        return os.path.join(self.base_dir, 'requests')

    @property
    def response_dir(self):
        return os.path.join(self.base_dir, 'responses')

    def msg_filename(self, user, request_id):
        return self.request_filename_format % (user, request_id)

    def request_file(self, user, request_id):
        return os.path.join(self.request_dir, self.msg_filename(user, request_id))

    def response_file(self, user, request_id):
        return os.path.join(self.response_dir, self.msg_filename(user, request_id))

    def create_msg_file(self, user, request_id, data, is_response=False):
        assert user and request_id and data and isinstance(data, dict)
        msg_file = self.response_file(user, request_id) if is_response else \
                   self.request_file(user, request_id)
        with open(msg_file, 'w') as fd:
            fd.write(yaml.dump(data))

    def msg_file_data(self, user, request_id, is_response=False):
        msg_file = self.response_file(user, request_id) if is_response else \
                   self.request_file(user, request_id)
        if os.path.exists(msg_file):
            with open(msg_file) as fd:
                data = yaml.load(fd.read())
            if data and isinstance(data, dict):  # expected data format
                return data
        return None

    def create_request(self, user, request_id, data):
        self.create_msg_file(user, request_id, data)
        logging.info('created request %s for %s' % (request_id, user))

    def create_response(self, request_id, status, response_data=None):
        assert status
        user = getpass.getuser()
        self.create_msg_file(user, request_id, {'status': status, 'data': response_data}, is_response=True)
        logging.info('created response %s with status %s for %s' % (request_id, status, user))

    def handle_requests(self):
        user = getpass.getuser()
        while True:  # keep handling requests indefinitely
            time.sleep(1)  # new request polling rate, in seconds
            for filename in os.listdir(self.request_dir):
                m = re.match('(.*)\.(.*)\.yaml', filename)
                if not m:  # not a valid request filename
                    continue
                [username, request_id] = m.groups()
                if username != user:  # not a request for this user
                    continue
                if os.path.exists(self.response_file(user, request_id)):
                    # request handling already started
                    # you may add here recovery code for request handling interrupted for whatever reason
                    continue
                msg_data = self.msg_file_data(user, request_id)
                if not msg_data:  # unexpected data format
                    continue

                logging.info('received request %s: %s' % (request_id, msg_data))

                # mark the request handling start
                self.create_response(request_id, 'in_progress')

                time.sleep(5)  # mock-up, replace with whatever request handling means

                # mark the request handling done
                self.create_response(request_id, 'done')  # you can add response data to the dict if needed

                logging.info('handled request %s, waiting for confirmation' % request_id)

                while True:  # wait for confirmation receipt before cleaning up
                    time.sleep(1)  # confirmation receipt polling rate, in seconds
                    if not os.path.exists(self.request_file(user, request_id)):
                        # the deletion of the request file is the confirmation receipt
                        logging.info('confirmation for request %s received, cleaning up' % request_id)
                        os.unlink(self.response_file(user, request_id))  # cleanup response file
                        break

    def execute_deployment(self, deployment_id, deployment_user, deployment_data):

        self.create_request(deployment_user, deployment_id, deployment_data)
        started = False
        while True:  # wait until it's done
            time.sleep(1)  # polling rate, in seconds
            msg_data = self.msg_file_data(deployment_user, deployment_id, is_response=True)
            if msg_data:
                status = msg_data.get('status')
                if status:
                    if not started:
                        logging.info('request %s handling started' % deployment_id)
                        started = True
                    if status == 'done':  # job completed
                        logging.info('request %s handling completed, cleaning up' % deployment_id)
                        # cleanup request file, which is the confirmation receipt
                        os.unlink(self.request_file(deployment_user, deployment_id))
                        break

def usage(err_msg, option_parser):
    if err_msg:
        logging.error('%s\n\n%s\n' % (err_msg, option_parser.format_help()))
    sys.exit(-1)

if __name__ == "__main__":
    import optparse

    logging.basicConfig(level=logging.DEBUG, format="%(levelname)5s  %(asctime)s %(filename)s:%(lineno)d] %(message)s")
    os.umask(022)

    p = optparse.OptionParser()
    p.add_option('-c', '--command', action='store', dest='command', choices=['deploy', 'handler'],
                 help='command/mode: <deploy|handler>, mandatory', default=None)
    p.add_option('-i', '--ID', action='store', dest='id',
                 help='deployment ID, mandatory for deploy command', default=None)
    p.add_option('-u', '--user', action='store', dest='user',
                 help='deployment user, mandatory for deploy command', default=None)
    p.add_option('-a', '--artifact', action='store', dest='artifact',
                 help='deployment artifact, mandatory for deploy command', default=None)

    opts, _ = p.parse_args()

    if not opts.command:
        usage('command is mandatory', p)

    mailbox = Mailbox()

    if opts.command == 'deploy':
        if not opts.id or not opts.user or not opts.artifact:
            usage('ID and user must be specified for deploy command', p)
        data = {'artifact': opts.artifact}
        mailbox.execute_deployment(opts.id, opts.user, data)

    elif opts.command == 'handler':
        mailbox.handle_requests()

The jenkins user driving the deployments, deployment info hacked to a string for this example - 'artifact' :

$ ./mailbox.py -c deploy -i 20 -u dancorn -a artifact
 INFO  2017-10-17 13:32:34,663 mailbox.py:49] created request 20 for dancorn
 INFO  2017-10-17 13:32:35,666 mailbox.py:109] request 20 handling started
 INFO  2017-10-17 13:32:40,678 mailbox.py:112] request 20 handling completed, cleaning up
$ ./mailbox.py -c deploy -i 123 -u dancorn -a artifact
 INFO  2017-10-17 13:33:32,359 mailbox.py:49] created request 123 for dancorn
 INFO  2017-10-17 13:33:33,362 mailbox.py:109] request 123 handling started
 INFO  2017-10-17 13:33:38,375 mailbox.py:112] request 123 handling completed, cleaning up
$

The apache users would launch the handler which in this example remains running (could be converted to a daemon, or a cron-driven approach):

$ ./mailbox.py -c handler
 INFO  2017-10-17 13:32:34,819 mailbox.py:77] received request 20: {'artifact': 'artifact'}
 INFO  2017-10-17 13:32:34,821 mailbox.py:55] created response 20 with status in_progress for dancorn
 INFO  2017-10-17 13:32:39,827 mailbox.py:55] created response 20 with status done for dancorn
 INFO  2017-10-17 13:32:39,827 mailbox.py:87] handled request 20, waiting for confirmation
 INFO  2017-10-17 13:32:40,828 mailbox.py:93] confirmation for request 20 received, cleaning up
 INFO  2017-10-17 13:33:32,888 mailbox.py:77] received request 123: {'artifact': 'artifact'}
 INFO  2017-10-17 13:33:32,889 mailbox.py:55] created response 123 with status in_progress for dancorn
 INFO  2017-10-17 13:33:37,891 mailbox.py:55] created response 123 with status done for dancorn
 INFO  2017-10-17 13:33:37,891 mailbox.py:87] handled request 123, waiting for confirmation
 INFO  2017-10-17 13:33:38,893 mailbox.py:93] confirmation for request 123 received, cleaning up


network ansible command error: "msg": "network os centos is not supported"

This error is due to invalid value of ansible_network_os:
The playbook you try setup is meant for network devices.
And since centos is not a network device OS the playbook will fail.
Here you can find supported network OS.
Ansible Documentation: ansible_network_os values
If you don't have any device running a supported network OS I recommend you to follow the regular user guide.
Ansible User Guide
My suggestion is to setup a easy playbook that will run on the environment you have.
Get the understanding of following concepts.
Control node
Managed nodes
Inventory
Modules
Tasks
Playbooks
Here is a short example.
This will only execute hostname do confirm it execute on expected host.
Let's say nicholas is your control node and rolly is the managed node.
On nicholas:
Create a inventory as before with your control node and managed node.
inventory.txt:
[local]
localhost ansible_connection=local

[managed_node]
rolly ansible_host=<rolly ip> ansible_user=<rolly user> ansible_ssh_pass=<rolly user password>

myplaybook.yml:
---
- name: This will get hostname localy on control node
  hosts: localhost
  tasks:
    - name: get hostname
      command: hostname
      register: result

    - name: print hostname
      debug:
          var: result

- name: This will get hostname remote on manged node
  hosts: rolly
  tasks:
    - name: get hostname
      command: hostname
      register: result

    - name: print hostname
      debug:
          var: result

And then execute on the control node:
ansible-playbook -i inventory.txt myplaybook.yml


Nginx routing setup within Docker container, request timing out with 502

You will notice in this upstream block:

upstream api {
    server server:5000;
}

I called it api but then I say server server:5000;. That was the problem because I originally had it as:

upstream server {
    server server:5000;
}

but then thought that server might be a reserved operator in the Nginx world so I changed it to api but did not change it inside the block.

So it should have been:

upstream api {
    server api:5000;
}

After changing it, the xhr pending and timing out 502 errors went away.


How to add a new column with custom value extracted from the job?

From what I can tell you can definitely customize the columns for a custom view (I can't find where to change it for the "All" view, but it's probably possible).  Unfortunately it doesn't look trivial to add a custom column without writing a plugin, for example the jacoco plugin:

https://github.com/jenkinsci/jacoco-plugin/blob/master/src/main/java/hudson/plugins/jacococoveragecolumn/JaCoCoColumn.java

Jenkins Java Doc:

http://javadoc.jenkins.io/hudson/views/ListViewColumn.html

Edit: This plugin might get you closer to what you're looking for:

https://wiki.jenkins-ci.org/display/JENKINS/Extra+Columns+Plugin


It looks like you tried to compile multiple policies into one.
Each original had a statement with its array of statements and you nested them.

This (untested) should work with each statement block within the array of Statement of the policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation",
        "s3:ListBucketMultipartUploads"
      ],
      "Resource": "arn:aws:s3:::<my bucket>",
      "Principal": {
        "AWS": [
          "arn:aws:iam::560184616970:user/<my username>"
        ]
      }
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:AbortMultipartUpload",
        "s3:DeleteObject",
        "s3:DeleteObjectVersion",
        "s3:GetObject",
        "s3:GetObjectAcl",
        "s3:GetObjectVersion",
        "s3:GetObjectVersionAcl",
        "s3:PutObject",
        "s3:PutObjectAcl",
        "s3:PutObjectAclVersion"
      ],
      "Resource": "arn:aws:s3:::<my-bucket>/*",
      "Principal": {
        "AWS": [
          "arn:aws:iam::560184616970:user/<my-username>"
        ]
      }
    },
    {
      "Effect": "Allow",
      "Action": "s3:ListAllMyBuckets",
      "Resource": "*"
    }
  ]
}

An alternative could be to use aws bucket policy generator


JAVA home give me wrong value on Jenkins node

Got response from SO forum: Is there a JAVA_HOME configured under Mange Jenkins -> Global Tool Configuration -> JDK? You may have to click on JDK installations to be able to see what's configured there.


Using SSH to open interactive session in ansible

You could try ansinble-console

It mostly feels like you just have a shell open, unless you use and ansible task name as your command, in which case prefixing it with shell <whatever you wanted to run> should work.


How do we encrypt credentials in Jenkins console output

Here's an example of using them in a pipeline safely.  In this case, they are injected as environment variables and their value should never have to be shown unless you explicitly choose to print out the variable content for some reason or other.

Environment variables are generally seen as the best way to store secrets at this point and are heavily relied on by many modern deployments (e.g. often used in kubernetes apps).

https://www.jenkins.io/doc/book/pipeline/jenkinsfile/#handling-credentials

Jenkinsfile (Declarative Pipeline)
pipeline {
    agent {
        // Define agent details here
    }
    environment {
        AWS_ACCESS_KEY_ID     = credentials('jenkins-aws-secret-key-id')
        AWS_SECRET_ACCESS_KEY = credentials('jenkins-aws-secret-access-key')
    }
    stages {
        stage('Example stage 1') {
            steps {
                //
            }
        }
        stage('Example stage 2') {
            steps {
                //
            }
        }
    }
}

You can also avoid Jenkins altogether by having your pipeline/etc retrieve them from some other tool like HashiCorp Vault, AWS SSM, Azure Vault, etc.


Having some trouble getting service running with Docker and Terraform

Seems like, because the application has security enabled, the health check was not able to hit the / endpoint and the health check was failing.  When I removed the health check block I was able to access the application.  I'll add a globally-open health check endpoint and that should fix the problem I think.


Is there a standard way to check if a web API is alive?

It would be nice if there was a more... official way. (: If this is not an official way how to check it? One could also say that if there are only POST's then why not using a POST?

Some services only have POST and these are checked by doing an automated POST every X minutes and check the output. If the POST fails then the monitor system will notify the team.


According to the documentation, memory storage is intended for testing purposes only and should never be used in production.  The only practical purpose I can think of for this might be state testing, such as with Serverspec or Inspec, to validate configurations.  Using memory storage for this purpose might be a little faster and use less disk storage.  I'm not sure those benefits are worth the effort to implement.


I believe your primary issue is that you have this group of users in the [Team Foundation]\Team Foundation Administrators group or possibly many users added with privileges to the Administration Console Users, which is unusual.  To adjust who is part of the Team Foundation Administrators group:

RDP to the server that hosts the application tier for your TFS installation
Open the TFS Administration Console
Highlight the Application Tier
Click the Group Membership link
Double-click [Team Foundation]\Team Foundation Administrators
Adjust membership of the group

To learn more about configuring administrators from the TFS Administration console, see Microsoft's documentation on the topic.


How do you monitor Node Micro services running inside of Docker Containers?

One commonly used solution(not container-specific) is to build a health check API within your service which tests all the functionality you want monitored (say availability of DBs and other dependencies) and the app itself, and returns some expected output (say < service >: < status >). You can then trigger alerts from a monitoring service like Nagios if this API does not return an ok for all the services. This will also fail if the microservice itself is unhealthy.

This approach also has the benefit of running a functional test of your service (by hitting an API end point).

This approach does not cover some edge cases though - eg. the microservice runs (but particular APIs fail).


Setting up Jenkins and git-server on same server

For me I use jenkins in docker with git plugin.

The Git plugin will give you the ability to check the Git server every specific period of time or you can use git hook from github or bitbucket to trigger the task when ever any change happened to the repo or the branch.

For deployment, most of time I use shell build step to execute some shell command on remote node.


Does RAM Come Into The Container vs VM Decision?

the benefit of a container starting up fast but they don't talk about heavyweight applications.

yes and no. The container only contains the app and runtime (e.g. JVM), but the app and runtime starts up with the same speed as when running on a VM. But a VM usually takes longer time to create and start since you also have to boot the operating system. So it depends on how you compare.
Running an app in a container does not do your app startup faster.

For this question I would like to know whether the RAM requirements of the processes running within them are a consideration when comparing VMs to Containers, or is there no difference between the two?

The app will use the same amount of resources when run as a container as on VM. But with containers it is easier to manage multiple instances - so you could consider that - this is usually done to achieve better availability and zero downtime deployments - however your app need to be stateless if you consider scaling out to multiple instances.


Seems like EFS is now supported on Fargate

Blog post
Reference


The goal can be achieved via several solutions. We can use service with awsvpc as network mode, so your service can get a real "A" DNS record and ENI interface, you must pay attention with this solution, ec2 instances has a limited number of ENI that can be attached, for a micro or small I think max 3.

Other solution more flexible is to use an internal NLB or custom proxy like HAproxy as ambassador, you run the proxy as service with awsvpc so he can get his "A" DNS record eg proxy.prod, all other services like Redis, Mongo, Elasticsearch ... can be configured as service with bridge networking mode so they will get an SRV record if configured with service discovery.

Finally you have to configure your proxy to redirect all traffic to desired service using port forwarding. HAproxy can resolve "SRV" records so you don't have to deal with that HAproxy will do it for you, you must just configure your app to use proxy.prod + port, for each service so:

Redis proxy.prod:6379
Mongo proxy.prod:27017
And so on...

Using this technique you can do a lot of amazing things, no matter where your proxy can lives, it can run as container in the same node with your apps or on a Fargate or in another node ...


How do you reference a code commit tag in IAM

I found what i needed to allow or the eqivalent StringNotEquals to deny useing the tag
aws:ResourceTag

"Condition": {
                "StringEquals": {
                    "aws:ResourceTag/Key_-1": "Value_-1"
                }
            }


What are reasons for using HashiCorp's packer to build docker images instead of using docker build?

Mainly the reason is to keep your image building steps intact if you get to move from docker to another image building system.

Packer does support a bunch of providers (builders in packer terminology), and changing the target "container" is just a matter of changing the builder (or using multiple builders in the same packer file), the build steps (provisionner step) are kept intact and will be the same if you build a docker image or an AWS ami for example (you can even build both at the same time).


I personally made the unwise choice of cramming five different apps into a single database about 20 years ago. I then had to deal with the repercussions of that. The bad news is, this is really hard. The good news is, modern tools make it easiser, but not easy.

So, first, absolutely get a single source of truth. You're using SQL Server, so there are a bunch of really good tools that will help you maintain your database in source control. One is provided by Microsoft. It's called SQL Server Data Tools. It only works inside Visual Studio. However, it's a solid tool. If you have an MSDN license to run VS, it's complete free. There are also 3rd party tools available for a cost. They're generally much more sophisticated (Microsoft doesn't give SSDT much love). I work for one of the vendors of such tools, Redgate Software.

Once you're on the path of maintaining the code in source control, the easy part is over. Now you have to figure out how to integrate the teams. Here's what we did. Every team got it's own branch of code (we did this in the old TFS, which sucked, you can use some variant of Git which will be much easier, not easy, just easier). This also means that each team had to have it's own copy of the database (which it sounds like what you're going for anyway). In fact, each developer got their own database and then  we had an integration testing database for each team. So the team worked on it's own, independent of the other teams.

Next, we created a master integration database. This is where the teams came together. We had a main branch for the code and teams would have to merge their code semi-regularly into this branch and then we could run automated tests to see if any team had broken the code or structures of other teams.

Then, you have to build a release process that ensures that any one team can't overwrite or supersede the others. This means that teams have to regularly merge from the main branch, changes from the other teams.

It requires quite a bit of discipline and some training to get this all set up. However, it's doable. I'd also suggest implementing a flow control software like Jenkins, Octopus, AWS DevOps or Azure DevOps to assist with automating all of it.

I've worked on tons and tons of documentation around these types of processes. Full disclosure, a lot of the info at that link includes our tooling. However, the processes are what you're looking for and you can find good information there. This is a free ebook on these processes I helped to write that's very product agnostic. Hopefully between these resources you get enough to help out.


Can large scale services use a public cloud managed database service?

Yes, it is certainly possible, but it entirely depends on the architectural design of the entire system. Separating application services into manageable components (i.e. microservices) is one approach to this. While Netflix as a whole is one application in terms of the user experience, it is comprised of many applications/services that provide a small fraction of the functionality needed to run Netflix as a whole.
In terms of a database, the same principal can be applied. Netflix doesn't manage one database for the entire application, each component utilizes it's own scalable database service.


Time series data stores in CI context - meaning?

Since time series datastores are indexed by timestamps they can be very useful in a CI/CD environment, depending on what kind of information they store, for things like:

incident analysis and management
service operation and SLAs monitoring
overall SDLC pipeline performance/efficiency analysis and improvements
resource capacity planning
budget and ROI computations


Declarative Jenkins Pipeline to Poll different branches?

One could use the PollSCM plugin.

https://jenkins.io/doc/book/pipeline/syntax/

triggers { pollSCM('H 4/* 0 0 1-5') }

  The pollSCM trigger is only available in Jenkins 2.22 or later.


Azure Devops Pipeline variables in Powershell

The issue is that you can't reference the changed value in the same task.
I created a new Powershell task and just called Write-Host "env:IP environment variable set to ($env:IP)" which output the IP address correctly.


How to Automate VM provisioning and setup using code?

First and foremost, there are several tools for better managing and provisioning of infrastructure such as Terraform.
You could create a pipeline to create things automatically as needed (based on triggers or manual start) on Jenkins or Gitlab, your code could be created using any language using the defined cloud provider SDK or API, or the. Terraform code as previously said. I'd use the Terraform since it's way easier to manage what is there, and sincronize the current cloud state.
Also, Ansible is great to setup machines, but Terraform can do it too.


How to setup priorities between pipelines that might be triggered simultaneously on Azure DevOps pipelines

It sounds like both of these pipelines are being triggered by the same source commit(s). If that's the case, you should be able to use CI trigger source path filters so that the builds only happen on commits to the relevant code.
If a developer does a single commit that is committing changes to both the main application and the package library (which would still trigger both pipelines regardless of the filter), you should consider promoting better code commit practices. Commits should be generally atomic to an appropriate context (i.e. don't commit to a stand-alone library and an app that consumes it together). Commit [and push if Git] the package changes first so they'll build and publish, then commit [and push] the main application. This will ensure that the trigger filters do what they should.
I don't know of a feature in ADO that allows you to serialize the pipeline execution order in the way you would like. As you suggested, the alternative of triggering one pipeline from another would kind of work but ultimately doesn't solve the core problem.
Alternatively, considering that "project N" changes a lot, and if it has that tight a relationship with the main project. Perhaps consuming it as a package is not appropriate. If the library is also to be consumed by other applications that have a more loose relationship, perhaps consume the dependency directly as code by the main project, but make its package publishing a part of the main application build instead of its own pipeline. (I've used this model for API applications that publish a data contracts library. The API application itself consumes the project directly as a project reference since the two are very tightly related, but others consume as a published package.)


As per compute/docs/troubleshooting/known-issues:

On Debian and Ubuntu based systems, including your local workstations,
you might encounter an error similar to the following example:
[...] The following signatures were invalid: XXXX [...]

To resolve this error, get the latest valid apt-key.gpg key file from https://packages.cloud.google.com by running the following command:
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -


Is there a policy that can limit the growth of an AWS S3 buckets total size?

Not as far as I know.

You can create Billing alerts to get notifications when AWS thinks that your monthly spend may be over a given threshold.


Exit a Makefile target called by parent without throwing an error

Makefile

all: test dosomething

test:
    @echo "hello"

dosomething:
    @if [ "a" = "a" ]; then\
        echo "world";\
        exit 0;\
    fi

Running: make will return:

hello
world


How to deterministic and reproducibly assign increasing port numbers in Ansible?

Disclaimer: I'm not using Ansible.

What I would do is use a random "predictable" number.
According to Ansible doc you can seed the random number generator:

  As of Ansible version 2.3, it’s also possible to initialize the random
  number generator from a seed. This way, you can create
  random-but-idempotent numbers:

  "{{ 59 |random(seed=inventory_hostname) }} * * * * root
  /script/from/cron"

So in your case for a port number (I assume unpriviledged) I would go for a variable with something like:

port="{{ 32767 |random(start=1024,seed=service_name) }}"

Max at 32767 to avoid clashing with any client initiated port (See Ephemeral port for the reason).


Error from server (Forbidden): error when retrieving current configuration of: Resource: "apps/v1, Resource=deployments",

Going from this, your ClusterRole isn't configured to allow access to deployments, and the ClusterRole you've listed isn't properly bound to your service account. You could configure it with something like I did below as a troubleshooting measure/to make sure you're able to properly configure permissions and rule out an issue with the serviceaccount's role bindings.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: gitlab-admin
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: gitlab-admin-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: deployment-reader
subjects:
- kind: ServiceAccount
  name: gitlab-admin
  namespace: kube-system

A new ClusterRole

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: deployment-reader
rules:
- apiGroups: ["extensions", "apps"]
  resources: ["deployments"]
  verbs: ["get", "watch", "list"]

Then to check to make sure the service account can properly access the resource in the default namespace you can check with the following command

kubectl get deployments --as system:serviceaccount:kube-system:gitlab-admin -n default


Connect two PCs to Azure vnet then RDP from one to the other

I have fixed this partially by updating the routes.txt file in the installation folder for the client app by adding an entry for the subnet used by the gateway to assign IP address to new logins.

While this now routes to the correct PC when RDPing, unfortunately a new IP is assigned to the PC every time it connects to the VPN, making it difficult to know the IP address when connecting.


If we are using Kubernetes do we need to implement HA on both the infrastructure and application level?

Your application requirements really do influence your options here, maybe add more detail about the app because this subject is very much about your application architecture too.

The application state is fixed, pod's are rebuilt as per the spec or source container that defines them and the replication sets take care of the deployment so I'd say the application from a Kubernetes perspective is already highly available by design and the reason why the documentation contains little coverage if any with regard to application level HA.

It's worth noting too that Kubernetes High Availability with kubeadm is still in its experimental stage and your clusters need to be running Kubernetes version 1.12 or later.

Google's Kubernetes Engine for instance; offers 'Regional Clusters' and these are spread over many zones in a region. This certainly increases your cluster availability. More here in GKE regions

The official word on the current state of Kubernetes HA cluster options are well documented here. https://kubernetes.io/docs/setup/independent/ha-topology/


nodes in a kubernetes cluster in azure unable to communicate with one another

With help from a Microsoft Engineer, we were able to resolve our issue.

The trick to it was that the node that could communicate with the outside world was always the one that the kubernetes tunnel pod was running on.  This indicated a problem with communication between nodes.  The traceroute indicated that this, with no network security group traffic restrictions, was bouncing between the node1 ip, to the load balancer ip, back to the node2 ip.

we had a security rule that restricted all traffic that was not on our specifically approved list.

One of the debug steps I had taken was to open all traffic between the load balancer ip to the load balancer ip.  However, this failed to consider the fact that the traffic was not from the load balancer to the load balancer, but from nodex to the load balancer then from the load balancer to nodey.

We added an inbound and outbound traffic rule that allowed traffic from the subnet ips that nodes can possibly be on to the load balancer, as well as within the same subnet for good meansure.  This resolved the issue.

If that's confusing, I can provide images that show the new rules and explain it a little clearer, but I'm hoping that is clear enough for anyone who experiences a similar issue.


I take for granted you are referring to the registry server released by docker.com. The documentation clearly states on the very first line:

  Before you can deploy a registry, you need to install Docker on the host

Since the registry is the main product of the docker/distribution opensource project, you could fork and see how you can modify this as a standalone install not requiring docker, but I have no clue if it is possible and that would probably end up being a bad idea.

There are other existing products you can install 'standalone' that will let you manage your own registry. But they all come with some degree of requirements (java, ruby...). I have experience with 2 free products I use daily:

Sonatype nexus3 oss is a multi format repository manager supporting docker registries (requires java).
gitlab-ce has support for container registries in each projects if you configure it (ruby required).

The latest one is probably an overkill if you only need a basic registry and not all of the git, wiki, etc. ressources.

I recently came across a fairly recent review article listing other products I have no experience with:

Harbor from VMware
Portus by Suse

It also mentions as a bonus the use of Openshift as a registry, but even minishift for something as light as possible will require to install docker.

All of these solutions can probably meet your 'native run' requirements, but they are all much more complex to install and maintain than simply poping-up a registry:2 container.