How to move disk image in the command line?

Docker is running in a VM on MacOS. One of your options is to put the entire VM on a tmpfs. You could use this shell script to create and mount the ramdisk: https://gist.github.com/koshigoe/822455

Here is a guide for resizing the machine. https://gist.github.com/stefanfoulis/5bd226b25fa0d4baedc4803fc002829e

The trick is to stop docker first, then you can make changes and move the

  ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2

wherever you like and symlink there from this location.


Monitoring tool to microservices on Kubernetes

AKS

https://github.com/MicrosoftDocs/azure-docs/blob/master/articles/aks/tutorial-kubernetes-monitor.md

Looks interesting. I have not tried it myself, but based on the more than 300 github stars this looks interesting to me. The readme contains screen shots how to configure it in Azure.


What is a secure way for storing AWS IAM credentials?

Well there are a lot of places to keep them secure

you can use something like hashicorp vault to keep those kind of secrets
you can use something like lastpass and my personal favorite is enpass

But first you need to define your needs so you can choose the right tool to use and integrate with your daily use


How to bring a docker data container to Kubernetes?

You'll want to look through the Kubernetes Volume documentation to see what options are available.

In particular look into the PersistentVolumeClaim.  This allows you to setup a shared data volume for your pods.  Here's an example snippet from a k8s deployment YAML using a volume:

spec:
      containers:
      - image: **yourdbcontainer**
        name: mysql-db
        env:
        - name: example
          value: example_value
        ports:
        - containerPort: 3306
          name: mysql-db
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /var/lib/mysql
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: mysql-pv-claim


The solution is to specify an assume_role statement:

provider "aws" {
  profile = "default"
  assume_role {
    role_arn = "arn:aws:iam::[ACCOUNT_ID]:role/terraform-test-role"
  }
}


Labels lets you as user query for particular objects; from the docs:

  Labels enable users to map their own organizational structures onto system objects in a loosely coupled fashion, without requiring clients to store these mappings.

For example if you use some kind of deployment tool like Spinnaker; the tool labels every single object that it touched so it knows about it - can manage it.

When you query for a label you can use equality-based operators (=, == and !=). K8s doc should be good starting point, alternatively check out this article on labels.


How to provide a user to access only a particular bucket in AWS S3?

Amazon's IAM roles generally grant a role access to a particular ARN (Amazon Resource Name). Amazon notes on their pages that for S3 a resource

  ...can be a bucket-name or a bucket-name/object-key.

They also provide a helpful example for doing just this which appears as follows:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket"],
      "Resource": ["arn:aws:s3:::test"]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:DeleteObject"
      ],
      "Resource": ["arn:aws:s3:::test/*"]
    }
  ]
}


Correct way to handle 'optional' array of resources in Terraform

Could you wrap the assignment to value using the try type conversion function?

output primary_LB_Subnet {
  value = try(aws_subnet.subnet_loadbalancer[0], null)
}


How to run a Windows container inside Windows Subsystem for Linux (WSL)?

C:\Users\romain.prevost>docker info  -f '{{.OSType}}/{{.Architecture}}'
'linux/x86_64'

Now, if you are running Docker for Windows, right click the Docker icon in your tray, and select "Switch to windows containers"

C:\Users\romain.prevost>docker info  -f '{{.OSType}}/{{.Architecture}}'
'windows/x86_64'

Voilà! Essentially, Docker for Windows includes two different engines : Linux in a VM, and Windows, natively on your windows kernel. But soon, there will be options to run Linux containers natively on the Windows kernel. Exciting times!


How to let build fail when docker image size exceeds certain threshold?

The output of docker images is really intended for humans, and it's not very parse-friendly. Instead, most Docker commands support a --format flag using Go templates. As you can see in the images documentation, you only really care about the .Size value, so you really want this:

docker images --format "{{.Size}}"

This just gives an output like this:

90.6 MB

If you have multiple images, they will each be on a separate line. I'm going to assume you will only ever have one line, because it vastly simplifies the next steps.

To get rid of the MB, we can simply use sed:

docker images --format "{{.Size}}" | sed "s/MB//g"
# => 90.6

But, you probably don't want this for your bash script. Most shells don't support floating point comparisons, so inputting 90.6 would just fail. Instead, let's use bc to truncate the number first:

docker images --format "{{.Size}}" | sed "s/MB/\/1/g" | bc
# => 90

Note that I've replaced the MB with a division by 1, which truncates in bc.

You can then pipe it into a shell script. I tested with this script:

VAR=$(cat)

if [ $VAR -gt "250" ]
    then
    echo "Image too large"
    exit 1
fi

Then run with:

docker images --format "{{.Size}}" | sed "s/MB/\/1/g" | bc | ./check_size.sh
# Exits with 0 (for my example)


jenkins - gitparam jobdsl - branchFilter

This is how I did:
gitParameter {
            // The name of the parameter.
            name('SELECTED_BRANCH')
            // The type of the list of parameters: Tag - list of all commit tags in repository - returns Tag Name Branch - list of all branch in repository - returns Branch Name Revision - list of all revision sha1 in repository followed by its author and date - returns Tag SHA1
            type('PT_BRANCH')
            // In my case this is a variable pointing to the gitlab hook variable
            defaultValue('$gitlabAfter')
            // A description that will be shown to the user later.
            description('Select a git the configuration you want to deploy')
            // Name of branch to look in.
            branch('')
            // Regex used to filter displayed branches.
            // Whatever is captured in the group will be the name used as the value of the SELECTED_BRANCH
            branchFilter('origin/(config\\/.*)')
            // This parameter is used to get tag from git.
            tagFilter('*')
            // Select how to sort the downloaded parameters.
            sortMode('NONE')
            // Which value is selected, after loaded parameters.
            selectedValue('NONE')
            // If in the task are defined multiple repositories, this option specifies which the repository is taken into account on getting data.
            useRepository("<my git url>.*")
            // When this option is enabled will show a text field.
            quickFilterEnabled(false)
            // Specify the number of items the list will display.
            // listSize(String value)
        }

As strange as it seems, some parameters shouldn't really be necessary, but the plugin requires them anyway, so that's why the config is verbose.
Also, pay attention that this one is gitParameter and not gitParam, yes, both exist, and they take different parameters.
For more information about the properties you can go here:
https://plugins.jenkins.io/git-parameter/


What's the benefit of using Nginx to serve a Flask API on AWS?

Unless you are using the AWS API Gateway, you'll still need some way of "serving" your Flask API.  You will also need to provision an EC2 instance and run something to serve your API.  Instead of running a windows EC2 instance with IIS you can save about 50% operating cost and use Nginx on a linux EC2 instance.  Additionally, Nginx can be used as a reverse proxy, and supports caching which you may find beneficial.

However, because you are using an Application Load Balancer and are looking into Gunicorn to serve the API you probably don't need Nginx, and it would just require more overhead for deployments if you are not going to setup any of the features in Nginx.


How to avoid interactive dialogs when running "apt-get upgrade -y" in Ubuntu 16.04 when packaging with Packer?

This sequence of commands works for me:

apt-get update
DEBIAN_FRONTEND=noninteractive apt-get upgrade -yq

So, DEBIAN_FRONTEND=noninteractive is correct but you also need the -q flag.

Source: https://github.com/moby/moby/issues/4032


Compared to just adding the user to the docker group the second links solution is not any more secure.

Note how that page still can launch with the --privileged flag, and that it is running unconfined.

unconfined_u:unconfined_r:unconfined_t

This means that the container can access all resources including the hosts disks and hardware.  It is security through obscurity, which is not security.  Anyone who knows how to use mknod or walk the /sys and /proc trees could easily compromise the host and all containers with zero logging.

The reality is that docker shifts both complexity and the security boundaries.  All users who can launch container or hit the API need to be restricted to trusted users in that security context.

--privileged disables all apparmor and selinux policies which is actually far less secure than a native package.

Namespaces are not a security function and depend on apparmor and selinux to enforce reasonable constraints.

Docker notes this on this page.

https://docs.docker.com/engine/security/security/#control-groups

'First of all, only trusted users should be allowed to control your Docker daemon.'

The security functions of docker/kube are not administrative boundaries like classical unix permissions, they are tools to prevent non-privileged containers from breaking out.

In a docker world the administrative boundary becomes the host, and the selection and segmentation required to isolate applications or users within that context needs to be applied at that boundary.

The benefits of this shift in complexity and responsibility generally outweigh the risks if implemented with those changes in mind.

TLDR

Docker API user == Sudo ALL user

Running a container with the --privlaged flag == running a web service as suid or as the root user.

Edited per the OP's request for additional information

The referenced issue with breakout int he OP's edit was an non uid0 privilege escalation.

Unfortunately, due to the need to perform root only actions Docker needs to enable some capabilities so that apt/dnf can install packages etc...

This need does pose a risk if production workloads are run in this default configuration and one should adopt the security principle of least privilege for production workloads.

--privileged disables apparmor/selinux and opens up capabilities

I am using ubuntu but it may be useful to work through the following steps. First start a default container with docker run -i --rm -t debian bash

From the parent host find the PID for bash using ps and note that the process is owned root.  If you look in /proc/$PID/status you will see the contexts it is running under.

# egrep '^Cap(Prm|Inh|Eff)' /proc/16026/status
 CapInh:    00000000a80425fb
 CapPrm:    00000000a80425fb
 CapEff:    00000000a80425fb

You will want to refer to man 7 man capabilities and /usr/include/linux/capability.h for better info but a summer is

CapInh = The Inherited capabilities (what docker provided)
CapPrm = The capabilities due to permissions (inside the container OS)
CapEff = The effective capabilities

You can decode these to human readable form by running:

$ capsh --decode=00000000a80425fb

Now do the same with $ docker run -i --rm --privileged -t debian bash and you will find that the effective capabilities are 0000003fffffffff

Also just do a dir /dev in both VM's and you will see just how much access a privileged container has.

By looking at /etc/apparmor.d/docker for apparmor systems or the lables in SElinux you will see the implications.

Going back to the principle of least privileges, I would ensure that my docker container is running process as a unprivileged user and with as few enabled capabilities as possible.

As an example, you can test this by running it first by dropping all caps.

$ docker run -i --rm -t --cap-drop=all -t debian bash

This can be validated through the /proc/$PID/status method above.

CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000

Note it is all zeros, from CapInh.  Then if I had to enable features for the application I would use --cap-add after the --cap-drop=all

Example, --cap-drop=all will break ping:

# ping -c 1 www.google.com
ping: Lacking privilege for raw socket.

So we can add the cap_net_raw cap. Docker expects the arguments with the cap_ prefix removed so the command is docker run -i --rm -t --cap-drop=all --cap-add=net_raw -t debian bash

# ping -c 1 www.google.com
PING www.google.com (216.58.216.164): 56 data bytes
64 bytes from 216.58.216.164: icmp_seq=0 ttl=54 time=4.779 ms

But be very careful adding them as an example.

cap_sys_module will allow a container to add or remove kernel modules from the parent host.

cap_sys_rawio will open memory and all block devices to attack

cap_sys_admin is super dangerous.

So in this case I would see if you can make things run in this context.

$ docker run -i --rm -t --cap-drop=all -u nobody:nogroup -t debian bash

Hopefully the robustness of apparmor and selinux profiles improves over time, but if that is not enough for your security needs you can look in there too.

Really, avoiding the --privileged flag and using principle of least privilege will make the most impact. Especially if you take advantage of the ephemeral nature of containers to keep packages up to date.

If you need more Red Hat covers some basic seccomp options here.

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_atomic_host/7/html/container_security_guide/linux_capabilities_and_seccomp


Volumes Not Mounting with Docker-Compose

If you let docker-compose manage the volumes for you, the volumes names in docker-compose.yml will not be the final ones created/used in docker. docker-compose will prepend the volumes names with the compose project name, which is by default the name of the folder holding the compose file. This is to allow separate projects to use the same names without conflict. For information, you will get the same naming convention for networks.

To get your data back correctly in your compose project, you have two choices.

Prefered in your case: Move data to automated volumes.

This will preserve the default compose volume management in your future runs.

stop any running container using any of those volumes
launch a dummy container not having mysql (e.g. busybox) and mount all 4 volumes in separate identifiable paths
empty all volumes created by docker-compose
copy all data from "docker only" volumes to "compose" volumes.
stop and recycle your dummy container.
docker-compose up -d and enjoy.
if all went good and smooth, delete the other unneeded volumes.

If you know what your doing: use external volumes

This will work as well but please check the documentation (Ctrl-f "Volume configuration reference") for details. Specifically, be aware that docker-compose up -d will no longer try to create the unexisting volumes and fire an error and stop if they don't exist.

Simply change you volume section at the end of your file to:

volumes:
  db-data-user:
    external:
      name: db-data-user
  db-data-system:
    external:
      name: db-data-system


Pause/Suspend guest in KVM without virsh and libvirtd

After long searching and experiments, the best option for me became qemu monitor. See https://nico.schottelius.org/blog/control-and-shutdown-qemu-kvm-vm-via-unix-socket/ for more details regarding flexible management guest by using qemu monitor.


depending of the registar and where your email server is you can. For example Ionos (ex 1and1) allow user to redirect any un-existing email to another email. This is done with the wildcard *@yourdomainname.ext


What are the differences between application virtualization, emulation layer, and container virtualization?

Container virtualization with Linux is a combination of kernel features, namespaces and cgroups, used to run an application in a sandbox. Each application has it's own filesystem, network stack, pid namespace, etc, but runs on the same OS kernel as the host. From the host you can see the processes running, but from inside the sandbox, all you can see is your application. It is designed for managing server side applications.

Application virtualization is very similar to container virtualization, and the lines are a bit fuzzy to me. In the context of something like Flatpak, it is using the same namespaces and cgroups you see with container tools like docker. The key difference is they have designed the tool for desktop use cases, meaning tools that are designed to be accessed with a local GUI, rather than running remotely as something like a web server.

Emulation is a way to run programs designed for another OS. You typically cannot run a Windows binary on a Linux system and vice versa, the OS kernels are very different, the binary formats are different, they're effectively speaking different languages. The emulation layer provides a translation to run non-native programs on the local machine without creating an entire virtual machine and running a separate OS. This is very different from the container and application virtualization where you run native OS applications on a shared OS kernel.


does boto3 have a way to get resource limits from a region?

I'm sure it does, otherwise it wouldn't be the module in use in:

https://awslimitchecker.readthedocs.io/en/latest/

From their setup.py installation script:

requires = [
    'boto3>=1.4.6',
    'botocore>=1.6.0',
    'termcolor>=1.1.0',
    'python-dateutil>=2.4.2',
    'versionfinder>=0.1.1',
    'pytz'
]

Thing is, maybe what you're trying to build is already built in some way or another. Try searching for libraries that has more or less the description of the tool you're trying to build, before starting.

Happy coding!


Is it possible to build Jenkins jobs without using build numbers?

It's not possible to avoid setting the BUILD_NUMBER environment variable, but you can avoid using it. Try using the Jenkins plugin Build Name Setter Plugin. It adds an option in your build configuration to set the build name to whatever you want, and you can ignore the BUILD_NUMBER environment variable this way.

Consider 1.250.5 versus 1.250.${BUILD_NUMBER}. The former will be constant no matter how many times you build and the latter will increment by one every time you build. It's also possible to use other macro values in between ${} to make more interesting dynamic build numbers.


Are all virtualization (OS/above-OS) utilities DevOps utilities?

I'd say you have touched another, far more fundamental question - what makes a utility a "DevOps" one? That is, how do you measure the "DevOpsness" of a tool and formalize facts in this context?

Possibly, my wish list for DevOpsness of a tool, virtualization tools or other, if they fulfill further criteria down on this list:

Open and interoperable API, both for operational and intelligence needs
Optimized workflows, see "developer experience"
Machine readable configuration as code
User community
Short release cycles
Certificate of software quality which is nowadays yet very ambigous


AWS SNS-SQS: Checking a subscription already exists

In the end I decided to go another way.
I don't think it was actually a good idea to modify things in that way when the worker boots. It can cause problems because if you have to do some emergency modification of the configuration those changes will be reverted the next time your worker instances restart.
Instead, on boot the worker checks if the queue exists. If the queue does not exist, it will create the queue and all its corresponding configuration including SNS subscriptions etc. It deliberately will only create the SNS subscription for the queue if also creating the queue.
This way, if some configuration has to be changed as an emergency in production, those changes won't be "reset" if the worker spins up new instances.
It is a bit annoying, but also safer I think.
Obviously, this is entirely personal opinion and what I went with in the end - but it seems to have served me relatively well so far. If anyone else has any better ideas or disagrees please do let me know.


Initial Configuration: Grafana and Prometheus inside docker and then AWS

Okay, it took time and help from someone else. I was going down the path of installing nginx, but that won't end up being necessary.

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    # <string, required> access mode. proxy or direct (Server or Browser in the UI). Required
    access: proxy
    orgId: 1
    url: http://localhost:9090
    isDefault: true

This is the end change. originally I had access: direct. As I understand it, that means access to the metrics is via the browser. Switching it to proxy means that Grafana will look up the data. Which means the URL doesn't need to be externally-useful.

To a newbie, none of this was obvious, so maybe my two days of pain will help someone out in the future.


How to copy library source code from other Git repo to an Android Gradle application GitLab build process?

What you need is not to checkout both projects side by side, but to add the library as a Git submodule in your application repository.
Then set up GitLab CI to checkout submodules before build.
This will not only make your CI work, but as an added benefit each commit from application repository will reference a specific commit from library repository, so  nobody will have to remember what commit to checkout in library repository to ensure the application builds correctly.
Edit:
To be clear, I'm not saying that composite builds (which is a proper name for your setup) are always bad, but I believe that they should be used for optional dependencies (like JetBrains did in their kotlin-native repository) or when developers need need to have an option of replacing dependency norally pulled from some Maven repository with locally build version (eg. to test if their app works with some changes in the library before submitting pull/merge request for the library) and not for everyday use.


Jenkins Artifactory Integration - Custom Maven plugin creating dynamic version/build

I figured out how to get this job to run which took a number of very specific steps:

1) The Maven3-Artifactory plugin will simply not do what I need it do here.  I still need this in my job though to actually execute my custom maven plugin from the project POM.  I configured the Build Environment for this plugin to NOT upload any build artifacts to Artifactory.

2) I added the Generic Artifactory plugin that has the responsibility of taking the zip artifact from the Maven plugin and uploading it to our Artifactory repo.  I struggled to define an upload FileSpec that worked correctly.  I ended up using the Legacy Patterns option where I could define the upload artifact and the repo location based on a number of Environment Variables that have been already defined for me previously (more info on this in future steps).  The Download Details are effectively empty.

3) The first Build Step is execution of the Maven build.

4) The second build step is execution of a custom shell that will parse the maven build artifact zip file for information that is needed to derive the version number.  It then injects this version number into a temporary property file in the root of the job workspace.

5) The third build step uses the Inject Environment Variables plugin to put the temporary property file into environment variables used by the Generic Artifactory Plugin Legacy Pattern.

This seems to work well as a solution for my situation.


Adding default external network in docker-compose

No that is not possible , it is called external because is created from outside the current docker-compose.yml you are working on


How can I use a local file for a nested Cloudformation stack?

I'm afraid you can't. The nested stack files must be in S3 as far as I know. You have a couple of options:

Have some automation in place (e.g. aws-cli + shell script, or better ansible) that when executed will upload all the templates to a designated S3 bucket and then execute the the CloudFormation deployment.
Use some templating engine like Ansible + Jinja or similar that can build a big CFN template from little snippets saved in separate files. With Jinja you can also use loops, lookup variables, etc, to build your final template file. Then again you can use the same Ansible playbook to deploy it. Or do it manually as it will only be a single file.

Hope that helps :)


I believe you will need to do this programmatically using the AWS SDK as described in Sending a Message (AWS SDKs). Here is a quote a from it:

  To send an SMS message by using one of AWS SDKs, use the action in
  that SDK that corresponds to the Publish request in the Amazon SNS
  API. With this request, you can send an SMS message directly to a
  phone number

  ...

  Sending a Message (AWS SDK for Java)

  The following example uses the publish method of the AmazonSNSClient
  class to send a message directly to a phone number:

public static void main(String[] args) {
        AmazonSNSClient snsClient = new AmazonSNSClient();
        String message = "My SMS message";
        String phoneNumber = "+1XXX5550100";
        Map<String, MessageAttributeValue> smsAttributes =
                new HashMap<String, MessageAttributeValue>();
        //<set SMS attributes>
        sendSMSMessage(snsClient, message, phoneNumber, smsAttributes);
}

public static void sendSMSMessage(AmazonSNSClient snsClient, String message,
      String phoneNumber, Map<String, MessageAttributeValue> smsAttributes) {
        PublishResult result = snsClient.publish(new PublishRequest()
                        .withMessage(message)
                        .withPhoneNumber(phoneNumber)
                        .withMessageAttributes(smsAttributes));
        System.out.println(result); // Prints the message ID.


How to add LDAP group for ssh access by ansible playbook?

You don't need to edit /etc/groups since the LDAP group already exists in your LDAP database. You want to let SSH know to use it, try using the AllowGroups setting in /etc/ssh/sshd_config.

From the manpage: "If specified, login is allowed only for users whose primary
group or supplementary group list matches one of the patterns.  Only group     names are valid; a numerical group ID is not recognized.  By default, login is allowed for all groups."

See this question for more information


Any suggestion for a provider to deploy a first Dockerized application?

You can launch a t2.micro instance in AWS and use it free for the first year. Check for the RAM and disk space and keep your docker image and container within those limits. You can either use AWS ECR as image registry or Docker Hub.


How to keep previous containers on docker-compose up?

docker-compose up will remove containers when:

The service name is the same, but has a change, in which case the old container is replaced with the new state of the service
You include the --remove-orphans option when running docker-compose

Otherwise, if you change the name of the service, and do not specify --remove-orphans, you will see "orphaned" containers still running.


TravisCI: Pandas version advanced (Jan 2021) numpy is now incompatible

This turned out to be a problem with pip determining the correct dependency: for some reason it thought that the pandas requirement was for numpy>=1.15
The error message
ERROR: pandas 1.2.0 has requirement numpy>=1.16.5 ...

was coming from pandas itself (upon import).
The fix is to upgrade to a newer version of pip (inside the script running on travisCI).  The newer version of pip is able to corrently detect that pandas needs a newer version of numpy.
Just put pip install --upgrade pip near the top of the script.
Not sure why TravisCI doesn't just use the latest pip by default.
Reference: https://travis-ci.community/t/11214


how to set automatic git pull from bitbucket and complie project on ec2 instance create event with auto scaling

You want to make use of user_data scripts.
It's basically a bootstrap shell script that gets executed on an instance during launch.
You can embed your app configuration script in there (either directly via bash, or have it call your configuration management software like Ansible or Chef).
Then, if you attach this script to a Launch Template or a Launch Configuration (which is an instance template required as part of an AWS ASG), it will be applied to the whole autoscaling group.
Then, when you have a scaling event, any new instances that come up in the ASG will run the script and deploy your code.
However, I would recommend you look into immutable infrastructure instead.  I.e. set up automation (i.e. via Packer) that builds your application code into your AMI, with user_data only used for actually starting the app.
Then, when you have a new build, have the same automation build a new AMI and replace your ASG (note you'll need to have your ASG created/deployed in a programmatic fashion.. I personally recommend Terraform).


System monitoring and reporting with Jenkins

having "ab"used Jenkins to be many things, the most sensible (IMHO) you can do is expose monitoring dashboards via build report page.
I think you should not even put Jenkins and monitoring under the same url, due to the "who will guard the guard" problem.

Also, nowadays monitoring tools are too versatile and have too many "faces" (a face for each characteristic user, and some specific users prefer to have their own personal "face" to monitoring)
About "reporting" I don't know what do you mean. if its reporting of builds statuses - READ ON.
Also there are other things to let user go through (logs, alerts, test reports)

So what I would suggest to address monitoring visibility is:

monitor your Jenkins in your monitoring :) (builds, statuses, etc.)
define monitoring url inside Jenkins (e.g. as Global Env Variable)
let's assume you have a job that deploys service $svc on environment $env. using Groovy you can generate:

generate a url to point at that monitoring service "dashboard" for that service in that env. let's call it serviceMonitorUrl
environmentMonitorUrl, and others
using addSummary and related stuff you can add to the build report a list of links to monitoring
you can decorate your build history column with "labels" pointing at branch, env, hash, whatever
your userdom will get build page with those links, and will be able to "click" them.

Having said that, there are Jenkins pipelines visualizations you can also use to visualize Jenkins jobs statuses, etc. e.g. this one.

Generally:
Jenkins is good for CI related scheduling.
But the truth is - CI should not be the focal point of users.
CI should be "transparent" until something is broken.
Thus making users "aware" of Jenkins (CI), and doing everything through it, unless business requirement is met by it (like company's product is being controlled/operated though Jenkins) - you should not do this. This kind of "knowledge" is not really serving the users.
They need to be able to see where to look for information easily, not be specific application's advanced users.

Think what happens when you decide to migrate from Jenkins to something else. you are hooked now.

(it isn't necessarily bad, just be aware of it)


AWS Automation Document Run Steps Against Different Servers Targeted By Tag

The issues were a lot, but the main one was not using Targets properly and then some YAML syntax problems.

---
description: "Example"
schemaVersion: "0.3"
assumeRole: "arn:aws:iam::{ID}:role/Example"
# parameters:
#   InstanceId:
#     type: "StringList"
#     description: "(Required) EC2 Instance(s) to start"
#   AutomationAssumeRole:
#     type: "String"
#     description: "(Optional) The ARN of the role that allows Automation to perform the actions on your behalf."
#     default: ""
mainSteps:
## STEP 1
- name: StartExampleServiceOnServer1
  action: aws:runCommand
  maxAttempts: 2
  inputs:
    DocumentName: AWS-RunPowerShellScript
    Targets:
    - Key: tag:Name
      Values:
      - ExampleOne-SomeService-Dev
    Parameters:
      commands:
      # Start service for SomeService
      - Start-Service -Name "ExampleOne"
      - Start-Sleep -s 30
  nextStep: ...


Connecting GitHub to Jenkins running on docker toolbox

Your configuration is wrong. Here you add http://192.168.99.100:8080/ is a local Jenkins URL. Github does not connect with the local URL. Also you give the wrong webhook-URL.

Please add your public IP of Jenkins in your webhook like http://35.24.128.75:8080/" And GitHub webhook payload url is like http://35.24.128.75:8080/github-webhook"

for Full webhook configuration please watch this video: https://www.youtube.com/watch?v=wsnq9uGAWkM&t=460s


How to handle services when the developers provide incorrectly in k8s

@Mkash, don't you know in advance what services you have in your services.yaml file?
Maintain a list in groovy, and see whether the input is in that list or not.
Something like:
availableServices = ["ew_ws_gateway", "ingress"]

services = ["a", "b", "c", "ew_ws_gateway"]

services.each { svc ->
    if (svc in availableServices) {
        println("Service exist" + svc)
    }
}

The other option is to change the parameter from type of text to multiple-choice possibly using https://plugins.jenkins.io/extended-choice-parameter/


Can I remove a docker image and still keep the container?

There are 2 really major points for Docker containers.

You make sure the app dependencies are versioned and encapsulated
inside the container. Container runs smoothly eveywhere, as long as you have the Docker daemon.
You have the capability to create/destroy a
container on any time, also known as ephemeral property. You could test easily, maintain N of them if needed ( Replica Sets in K8S for example are doing just that, Docker Swarm uses services to maintain this number as well ).

The official Docker docs defines an image as

  An image is a read-only template with instructions for creating a
  Docker container

You can't remove an image, because the container relies on it.

docker image rm ubuntu:20.04
Error response from daemon: conflict: unable to remove repository reference "ubuntu:20.04" (must force) - container ecd06b4eb4bf is using its referenced image

Then you might force the process

 docker image rm -f ubuntu:20.04
Untagged: ubuntu:20.04
Untagged: ubuntu@sha256:747d2dbbaaee995098c9792d99bd333c6783ce56150d1b11e333bbceed5c54d7
Untagged: ubuntu@sha256:8bce67040cd0ae39e0beb55bcb976a824d9966d2ac8d2e4bf6119b45505cee64

But that just untags the image, it's still here.

docker image ls
REPOSITORY             TAG                                        IMAGE ID            CREATED             SIZE
<none>                 <none>                                     1d622ef86b13        3 weeks ago         73.9MB

So to your question, the best practice is to only remove images that are not being used by any containers. If you maintain 2-5 versions of your app, you could safely remove the others. Also you schould strive for minimum size of each image, even in total that schould not bother your available space that much.


How can I manage changes in desired capacity due to scaling policies in AutoScaling Group state?

The desired_capacity in Terraform is marked in the documentation as optional. So with a proper min_size value, Terraform can wait until the minimum capacity is reached before continuing.

The above, with the addition of scale policy can effectively manage capacity without being specific about desired_capacity in your Terraform code. This will prevent you from having Terraform override the capacity in place at the moment of execution.


Consul Helm chart: Connect Injector service not working

I was reading the documentation to learn more about the admission controllers, when I read the prerequisites; the pertinent one being

  Ensure that MutatingAdmissionWebhook and ValidatingAdmissionWebhook
  admission controllers are enabled

Per the docs there is a recommended set of controllers to use:

  --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,Priority,ResourceQuota

I manage K8s clusters with Rancher in non-test environments. By default, Rancher only enables a few of these controllers:

  NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,DefaultTolerationSeconds,ResourceQuota

When I upgraded the cluster, this did not get enabled as I suspected with just the version upgrade. After updating the Rancher K8s configuration and applying the changes, I successfully spun up the test client/server deployments which successfully injected the envoy containers and registered the services in Consul.

TL;DR: RTFM


What is the best Windows Infrastructure to run Adobe After Effects like AWS Lambda?

In general a platform-as-a-service solution runs the application code on top of a provider-supplied sandbox infrastructure which complements your app code so that together they implement the complete app functionality. But this puts specific requirements (often quite restrictive) in front of the application code, for example:

a limited choice of (typically high-level) programming languages
certain pieces of mandatory functionality in your app code (like health checks, etc)
certain limitations of what the app can do (for example not allowing writes to the local filesystem)

So when looking for a platform-as-a-service solution you need to be looking from both the app perspective and the sandbox perspective.

Now your case brings additional requirements to the game:

running a 3rd party app which you cannot modify, so you need to find a PaaS offering whose requirements are met with the application as-is, unmodified
a Windows-based sandbox - rare, most providers use open-source infrastructure, I suspect primarily because of the licensing terms. Your highest chance of finding such offering would be on Azure, but not necessarily the only one
AE licensing terms - how many AE instances can you run?

Due to the extra requirement on the sandbox a infrastructure-as-a-service solution would be, IMHO, a better fit for your case (you're actually using it presently, maybe just scaling it would be a much simpler solution for you?).

AFAIK the only offering (outside of Azure, with which I'm not familiar) that comes very close to an infrastructure-as-a-service but still is, in a way, a platform-as-a-service is Google App Engine Flexible Environment:

meeting the app requirements might be possible by actually writing an app implementing a rendering service which uses AE under the hood, as a dependency
it supports (docker-based) custom runtimes, you might be able to build one with AE/Windows (maybe how it's done for ASP.NET apps would help)


If we are using containers do we still need to worry about configuration management at an infrastructure level?

Welcome to containers, a very exciting direction indeed.

Software configuration management tools such as the ones you mention do cater for containers in many ways and if you're already familiar with Terraform then you'll probably already know this.

Kubernetes for instance can manage deployments and replication sets (of containers; 'pods') and nodes much like Puppet might manage virtual machines and micro-services.

I would say I personally use Puppet less in the container world but believe configuration management is still necessary in a container infrastructure though you may change the way you approach it with regard to the native tools that come along with containers as opposed to Puppet perhaps.

A Dockerfile is fairly self explanatory to satisfy the infrastructure as code perspective that Puppet delivers; and if you then get into Kubernetes clusters and nodes the same could be said about YAML files too.

Containers are bundles of repeatable infrastructure. I find this stack overflow answer supports my answer too; it certainly helped me in the past.

https://stackoverflow.com/a/41472271/1524645


How do you change a secret that half of your production applications use right now?

This question ties in heavily with your other question with where the secrets are set and where they retrieved from.

I assume your question is asking about 'canary deployments' where you change the config for only a small portion of your apps, to test things out before deploying everywhere.

If you follow the best practice of using the 12 Factor app's recommendation of 'environment variables', then changing secrets becomes very easy. At the very worst, you could log into the production server, update the ENV VAR, and then restart your app manually. In Kubernetes, this is even easier, as you can edit the ENV variables from the UI, and then restart the pod.

However, you don't want to be doing that manual process if you can avoid it. If your Configuration Management tool sets the ENV VAR, then you can tell it to update a specific group of servers only. If your build server injects the ENV vars, and it supports canary deployments, then you can easily just do a canary deployment like normal.


docker mysql container not running suddenly

Without seeing the specific error it looks like the container's runtime process is either crashing or exiting.  When a docker container's runtime process exits the container stops.
 You can get more information about the specific error by using the docker logs command:

docker logs 92455a993a54


Connect Docker container to both host and internal bridge network

Docker does not allow to connect a container to the host network and any other Docker bridge network at the same time. I will try to illustrate the reason with an example:

Let us think of a container C1. Hypothetically, C1 would be connected to the host network (--net=host) and a Docker bridge network Br1 (--net=Br1).
A second container, let us say C2, is connected to Br1.

With the above setup, my guess is that the host network is visible from C2, and I suppose this is the reason why Docker automatically prevents us from unintentionally exposing the host network to non-host-specified containers.

That being said, if we have a set of containers, and we want all of them to be interconnected, with just a single container having access to the host network, my approach would be:

[C2,...,CN] are connected to a user-defined Docker bridge Br1 (--net=Br1)
C1 is connected to the host network (--net=host)
C1 exposes a port in order to be accessible from the rest of containers

EDIT: We would still have to adapt iptables policies in such a way that C1 can be reached from the rest of containers (see https://docs.docker.com/network/iptables/)


1 second system engineering work cycles: educated workflow setup for Docker-driven development

In some cases, depending on the actual configurations being targeted, it might be possible to try the effect of various changes live, using other automation tools, and only update the docker images (which acounts for a big chunk of that waiting time) after the bulk of the config changes are eliminated.

Stabilizing first the common/base docker image used across most/all of the candidate variants and sharing it would also help.


How to have multiple git repos in an Azure DevOps project?

Following works for me:

Click on your Project and then [Repos] in the left hand menu.
Along the top, you'll see the bread-crumbs: Organization / Project / Repos / Files / {NameOfCurrentRepo}
Click the down arrow next to that and you'll see the option to add a new Repository

If that's not available, then it must be a permissions setting that I'm not aware of.


Terraform - gitlab provider does not start

GitHub is one of the providers not yet included in the Terraform registry - see this diff to the documentation. This appears to be a temporary situation:

The following providers will be published on the Terraform Registry soon, but aren't quite ready. Until they're published, their documentation is available at the links below:

I can't tell from the release notes whether this is only in 0.14.x, but I can confirm that the "old" (i.e. current at the time of writing) github provider works with 0.12.x


How to automate installing an ISO in a VM?

If you're into RHEL you could look into running oVirt ( https://ovirt.org/ ). This will enable you to manage host machines, VMs and virtually anything in between through Ansible.

What you need is oVirt on top of CentOS or similar, on a baremetal machine. This/these machine(s) will host your new VMs and your so-called Hosted Engine (i.e. the oVirt virtualization management engine run from within a VM running on the hosts themselves to ensure high availability and flexibility), and then you just need Ansible and the oVirt Python SDK.

Once you have this, it's "as simple as" creating Ansible playbooks and using the oVirt Ansible modules https://docs.ansible.com/ansible/latest/modules/list_of_cloud_modules.html#ovirt


How to hot deploy to Tomcat with Ansible

At the bottom line I found this method as stable and reliable, in case you want to deploy via ansible.

### This play deploys a webapp war to an tomcat server

- name: Copy the war file {{ webapp_name }}.war as {{ webapp_name }}.war.tmp to {{ webapp_destination_folder }}
  win_copy:
    src: files/{{ webapp_name }}.war
    dest: "{{ webapp_destination_folder }}\\{{ webapp_name }}.war.tmp"
    force: yes

- name: Remove current war file {{ webapp_destination_folder }}\{{ webapp_name }}.war
  win_file:
    path: "{{ webapp_destination_folder }}\\{{ webapp_name }}.war"
    state: absent

- name: Wait until the webapp folder {{ webapp_destination_folder }}\{{ webapp_name }} is deleted
  win_wait_for:
    path: "{{ webapp_destination_folder }}\\{{ webapp_name }}xxx"
    state: absent
  register: folder_info

- name: rename the {{ webapp_name }}.war.tmp to {{ webapp_name }}.war
  win_command: "cmd.exe /c rename {{ webapp_destination_folder }}\\{{ webapp_name }}.war.tmp {{ webapp_name }}.war"
  register: cmd_result


Multiple paths with different expiry time in gitlab-ci runners

This has been answered over on SO using a work around seeing as it doesn't seem possible according to the documents.

Basically, this can be done in 3 stages.

Stage 1: Build and store all artifacts.

build_stage:
  script:
    - build
  artifacts:
    paths:
    - build/*.dmg
    - build/test1
    - build/test2
    - build/test3
    expire_in: 15 mins

Stage 2.1: Do the next official stage of the job (ie run tests in my scenario) using the artifacts build/test1, build/test2 and build/test3.

test_stage:
  script:
    - test
  dependencies:
  - build

Stage 2.2: Running concurrently with Stage 2.1 you can just have an empty job but sets a new artifact expiry date.

overwrite_artifact_stage:
  script:
    - echo 'saving artifact'
  artifacts:
    paths:
    - build/*.dmg
    expire_in: 1 week


Azure DevOps - how to create a 2nd Microsoft-hosted Agent Pool

This is a good question because you would assume you could do what you are after, but as far as I know you can't yet. From the docs on sharing of parallel jobs across projects in a collection

  Currently, there isn't a way to partition or dedicate parallel job capacity to a specific project or agent pool.

I considered trying to do buckets via capabilities on a subset of my agents, and then companion demands in the pipelines, but that will not work either because hosted pipelines don't work with demands.

In my case we had some greedy pipelines that fan out into multiple parallel jobs, that we had to update to run sequentially with dependsOn for each job. I think if your case is relatively simple you could potentially check the queue length or number of running builds in a first stage but that could get complex and might be tough to scale.

Hopefully this feature gets added soon. I will look to see if I can find one to vote up on the developer community forums. I am also interested in other ideas on how to solve for this issue!

Edit found a feature vote on it here


How to automatically update Docker containers in development?

An approach I use in both prod and dev is to run a webhook server ( this one is excellent https://github.com/adnanh/webhook ) which is subscribed to listen to the github.com  git push events which are published by github

... then when someone does a push to github.com the webhook launches a rebuild script to issue git pull then recomple the code and issue docker build xxx and docker push yyy  and docker-compose -f my-file-docker-compose.yaml up   ...  this all happens on auto pilot ...  where I have a running webhook server on a vps box for a given git branch to rebuild its toy domain

this took time to craft however its a life saver


Re-initialize etcd-cluster completely (on coreos)

I managed to fix it. First I disabled and deleted all etcd2-related units. Then I created a new dropin to the etcd-member.service with the following content:

[Service]
Environment="ETCD_OPTS=--name cluster00 \
  --discovery https://discovery.etcd.io/<MY_ID> \
  --listen-client-urls http://<NODE_IP>:2379,http://127.0.0.1:2379 \
  --listen-peer-urls http://<NODE_IP>:2380 \
  --advertise-client-urls http://<NODE_IP>:2379 \
  --initial-advertise-peer-urls http://<NODE_IP>:2380"

I generated a new discovery id for the cluster.

Then I teared down the old cluster completely, by deleting all content of /var/lib/etcd/* and the directory /var/lib/etcd2 completely and (re-)started the etcd-member service.

Be aware, that I don't have ssl enabled at this point!


Automatic Docker Hub builds from GitLAB, not github

This feature is not supported as the issue in the link says.

They promised it for late 2016 but now at 2017 the issue is still open.

One comment was to look for gitlab CI, which might be my best guess of workaround  for now, waiting the Docker Hub version to come.


Is running docker-compose not supported in BitBucket?

Here is my solution: (bitbucket-pipelines.yml)

pipelines:
  default:
    - step:
        script:
          - export DOCKER_COMPOSE_VERSION=1.18.0
          - export DOCKER_COMPOSE_URL=https://github.com/docker/compose/releases/download/${DOCKER_COMPOSE_VERSION}/docker-compose-$(uname -s)-$(uname -m)
          - curl -L $DOCKER_COMPOSE_URL > docker-compose
          - chmod +x docker-compose
          - mv docker-compose /usr/local/bin
          - docker-compose build --force-rm --no-cache --pull


Sounds like the docker image you are using doesn't have the correct name, or is stored in a private repository which requires login.

Please post more information


Can I build docker image containing ARM executable when running docker build on non-ARM architecture?

I've not tried this myself, but it seems that you can use docker buildx to build for multiple architectures: https://www.docker.com/blog/multi-arch-images/.


What reasons should I NOT use a private DNS zone and CNAME services in my AWS/K8s environment?

It seems a reasonable idea to me.
There migt be a few things you might want to check tho :
This article or this one. For example :

the fact the that a private hosted zone is associated with a VPC
there is some DNS hostnames and DNS resolution parameters on the VPC settings

enableDnsHostnames : Indicates whether instances with public IP addresses get corresponding public DNS hostnames
enableDnsSupport : Indicates whether the DNS resolution is supported.

Only VPC resolver will resolve the private records (subnet range +2 for the ip of the VPC resolver, search for reserved ip here)

take care you didn't configure custom resolver and so wont be able to access your private records
think about the fact that you won't be able to resolve those records from outside the VPC (office for example) unless you manage to plug to the VPC resolver somehow

You could also give a look at Alias record ?
Could give advantages like :

create record that has the same name as the hosted zone
not being charged for the queries (in opposite to CNAMEs)

These are few things I can think off right now. Hope this helps.
Cheers.


How to synchronize deployments (especially of database object changes) on multiple environments

I would integrate the schema management into the application itself (or along with it).

Any change to the schema should be committed along the application code (and hence tagged also).

There's already a bunch of possibilities listed in this question: What practices or tools enable Continuous Deployment of Databases

With this kind of tools, using an in memory database like h2 or a central database and having the application check the schema version (stored in the DB) when started allow to get an updated database schema at application startup.

This will not solve your discipline problem and may create new ones, as this usually comes with a strict versioning of the schema, enforcing the version can be done with pre-commit hooks but that's a hard work.


How to fail stack when EC2 spot is refused?

You should use TimeoutDurationMinutes in your code while requesting for spot instances via cloudformation.

For more details: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-elasticmapreduce-instancefleetconfig-spotprovisioningspecification.html


What artifact managers are available in GCP?

If the artifacts in question are docker images then the recommended artifact management solution is the Container Registry, well integrated with other GCP products producing and/or using such images.

AFAIK they don't have a real artifact manager for other kinds of artifacts, they suggest the rather general purpose Cloud Storage for that. You can find an example of such use in the Quickstart for Go guide. Most likely not what you're looking for:

artifacts:
  objects:
    location: 'gs://[BUCKET_NAME]/'
    paths: ['hello']


You should add the correct zone attribute for the host and the service, so they will propagate to the client node.

Add "zone": "client.com" to the api requests.


Terraform creating single SG but multiple CIDR blocks

Figured out the answer, fourtinately for me I had missed the example in the blog post about terraform 0.12 linked below

Here is my final (working how I want) security group

resource "aws_security_group" "mongo" {
  name        = "Mongo"
  description = "Allow mongo traffic"
  vpc_id      = aws_vpc.Main_VPC.id

  ingress {
    from_port   = 27017
    to_port     = 27017
    protocol    = "tcp"
    cidr_blocks = [
      for num in var.private_subnet_numbers:
      cidrsubnet(aws_vpc.Main_VPC.cidr_block, 8, num)
    ]
  }

  tags = {
    Name = "Mongo"
  }
}

Link to blog post about foreach - https://www.hashicorp.com/blog/hashicorp-terraform-0-12-preview-for-and-for-each/


Disabling of docker build cache - common practice for large organizations?

I guess each organization does what is best for them, but one of Docker's best practices is to leverage build cache.
Applied to a large company, I'd say that if they would use Docker caching (correctly) and save a couple of seconds/minutes on every build. They would, given their size, save many hours/days of Engineering time, which amounts to a lot of money. Also, when a build takes too long, it breaks the flow of the individual, which means more context switching on the side of the engineer, which usually leads to less productivity.


Ansible Playbook Localhost Invocation Working Directory

You can use env lookup plugins. Plugins are always evaluated in the context of "parent" ansible process on the control host.

- debug:
    msg: "{{ lookup('env', 'PWD') }}"

More lookup plugins can be found here: http://docs.ansible.com/ansible/latest/user_guide/playbooks_lookups.html


What is the way to update images in a configuration git repository when doing gitops?

2) App repo's CI commits into Configuration repo with newly built image tags.

Yes, this makes sense. Instead of using image tags you can use image digest, since the digest is generated from content, it is immutable, while tags may be updated (intentionally or unintentionally).

This manifest update can be done with multiple tools:

Using yq

yq write --inplace  deployment.yaml 'spec.template.spec.containers(name==myappname).image' gcr.io/my-image-repository/myappname:labelOrDigest

Or using a newer tool; kpt with the ktp set command.


Jenkins Docker Behind Reverse Proxy Adds ":80" to the URL

If it's a brand new Jenkins, you can try creating an SSH tunnel directly into the container with port 8080 and checking if the problem also appears there. That will at least allow you to check if it's Jenkins redirecting you or Nginx.

There you can also try adjusting "Jenkins URL" in "Manage Jenkins" -> "Configure System".

Alternatively, check  jenkins_home/jenkins.model.JenkinsLocationConfiguration.xml. jenkinsUrl tag is also present there.


Q: "Override certain parameters that pass them into the include_role task."
A: The parameter tasks_from serves this use-case. For example, create the task file
shell> cat roles/git_cached/tasks/git.yml
- git:
    repo: "{{ params.repo }}"
    dest: "{{ params.dest }}"
    recursive: "{{ params.recursive|default(ommit) }}"
    depth: "{{ params.depth|default('1') }}"

Then "call" this task
    - name: Download something via Git
      include_role:
        name: git_cached
        tasks_from: git.yml
      vars:
        params:
          repo: https://somerepo
          dest: /some/path

You might want to complete the task with all git parameters:

The parameters repo and dest are required.

In some parameters, like recursive, the default values are defined. You can either omit the parameter, in which case the default value will be used, or define your own default.

Some parameters, like depth, don't have defaults.  Also here you can either omit the parameter, in which case the parameter won't be used or define your own default.

It's a good idea to "namespace" the variables.

It's a good idea to put the default variables into roles/git_cached/defaults/main.yml


How to ensure Nodes are killed when Job is finished using Cluster Autoscaler in Kubernates

When a Job completes, no more Pods are created, but the Pods are not deleted either.
https://kubernetes.io/docs/concepts/workloads/controllers/job/#job-termination-and-cleanup

clear it by another cronjob.
use operator to wrapper it and add remove mechanism in the operator.


Merging Git Database Commits from One Branch to Another without Cherrypicking

I read in Git, Cherry picking is bad.

Be careful of such blanket statements. They take on a life of their one, and after a while you have a lot of myths in your team (or workplace) about no cherry-picking, no re-basing, no xyz. These are all just tools; there are times to use them, and times to not use them. Nothing is generally bad.

  Sometimes , we do not move all Release items into Master, many reasons:, delay schedule, conduct more testing, late issues. What is the proper Git Devops strategy to only move certain items into Master?

Adam Dymitruk's Branch per Feature works splendidly for that. In short:

There are these branches:

1 master branch
1 qa branch
n feature-123 branches

The general principles:

Every branch starts at master.
Only feature-123 branches receive regular commits.
qa is created by starting fresh from master, and then merging any feature-123 branches that should be in testing. Whenever qa needs to be changed, it is created fresh again, there are no other commits into it. This happens very frequently.
When qa has been released/deployed, it is simply renamed to master and replaces the previous one. At this point, all remaining feature-123 branches are rebased onto the new master. The git rerere cache is used to avoid having to do repeated conflict resolution.

This scheme works great if you have developers who can stomach the rebases. You frequently find people who think they are bad, or who simply do not allow it - then you're out of look obviously. But if you get it to run in your team, it is very smooth, and ultra flexible. You literally never ever need to "back a commit out" or do cherry-picking to revert a commit, or whatever.

Read Adam's page, it explains this in more detail, but basically that's about it. You'll need to create your own tooling for the individual steps (for example, to pull the feature branches which are supposed to go into qa from your Jira instead of manually doing that), but that's not too hard.

Addendum:

Trivially, you can have more than one qa-type branch at any point in time. For example, you will likely start with one that always has all features which are deemed close to "done" (but which may not have successfully passed test yet); and maybe another one (let's call it rc - release candidate) which contains a subset of that for the very next release.
Adam also describes a ci branch; it is nice, but optional (we don't use it at all, and are not missing it).


Is it required to move a domain to Google DNS in order to use certbot in kubernetes on Google Cloud platform?

@euank indicated that it is not required to create a DNS record on GCP in order to make it work.

https://github.com/jetstack/cert-manager/issues/399


What's the difference between travis-ci.org and travis-ci.com?

Travis CI originally created two separated platforms to differentiate between private repos / paid (travis-ci.com) and public repos / free (travis-ci.org).
However, as of May 2018, new users and projects (both private and public) should only use travis-ci.com (see this blog post). Note that travis-ci.org will be closed down completely on December 31st, 2020 (see this newsletter).
Although still in beta—which is a bit weird since travis-ci.org will be shut down soon—Travis CI provides a well-documented migration guide.


The registered variable smm_output is a dict object and you can correctly dive into it. But you cannot use smm_output.results.stdout_lines because smm_output.results is a list. You need to iterate over it or filter the first element of results. So, if you know its only one element in the list, you should use smm_output.results.0.stdout_lines.


What are the advantages of using AWS nameservers as opposed to servers provided by other registrars?

There are two parts to owning a domain. The registration, and the hosting of the DNS entries. As you are transferring the domain registration, AWS needs to know whose service do you want to use for managing your DNS host names and other settings.
As a sensible default, select option (1) as this will keep everything managed with the current registrar / hosting company and so nothing will break (at this point in time). When your hosting expires with this company, its quite likely they will drop all of your DNS settings as well, in which case you will need to have migrated somewhere else before this point (options 2 and 3)
However, if you are using AWS services you really should be managing the DNS settings in AWS and create a Route 53 hosted zone. This provides better integration with the AWS environment, and allows you to use some of the "AWS aware" aspects of Route53 for your infrastructure.  Assuming that you initially selected option 1, you should then set up a parallel zone in Route53, verify all of the DNS records match (except NS settings for the domain) and then update your Domain registration to use option 2. Changing this setting can take 72 hours to roll across the internet. In my experience, usually 90% of the internet knows in under 6 hours.
Option 3 means you (or someone else) is managing the Domain DNS master server for your domain. There may be situations where this makes sense, but they are rare.


Why is environment variable mangled in docker compose environment file

I had a similar problem and this worked for me without it feeling like a hack...

docker-compose.yml:

version: '3'
services:
   web:
      build .
      environment:
      - GUNICORN_CMD_ARGS=--workers=0 --bind=0.0.0.0:8000 --timeout=10

Note: add env GUNICORN_CMD_ARGS without quotes

Dockerfile:

FROM python:3.6-slim
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt
ENV GUNICORN_CMD_ARGS="--bind=0.0.0.0:8000 --workers=3"
CMD gunicorn app:app

Note: run CMD gunicorn app:app without passing options via array or quotes


Why does Jenkins safeRestart not wait for job to finish?

Jenkins pipeline jobs are designed to be serializable to disk, to increase durability (their words not mine). Therefore they are considered safe to be suspended. The only things you can do is either wrap your stages in functions with @NonCPS decorator or using properties make the jobs non-resumable after restart, but this will cause them to not be resumed after restart.

Also, restarting jenkins is not really a thing that happens all the time, so it might not be worth the effort to protect against.


How to publish Jenkins job icons externally?

The "Embeddable Build Status Plugin" will do exactly that.

Note: This requires your Jenkins server to be accessible from the internet if you're using GitHub because they cache all images in their CDN.


Ansible - Handler not found on first attempt

Name of the handler "Restart NTP" does not fit the handler notified "restart ntp". This is the reason of the error. On the second run the configuration file /etc/ntp.conf did not change ("changed": false). Therefor the handler was not notified, was not looked for and no error was reported.


VPN container providing access to host's LAN?

I am accessing my home network with a VPN running in a docker container in my home server.

That sounds similar to what you want to do.

Here's how I configured it (using this docker image - note that the documentation of the docker image should be enough)

Use a “convenience” environment variable to store the path to your persistent storage location that will be bind-mounted to the container.

OVPN_DATA="/n7wings/openvpn/"

Run an ephemeral instance (–rm) of the image to initialize the data directory of the container (ovpn-host should be the hostname of your openvpn server)

docker run -v $OVPN_DATA:/etc/openvpn --rm kylemanna/openvpn ovpn_genconfig -u udp://ovpn-host

Run an interactive ephemeral instance of the image to generate the opevnpn CA certificate and server key (you will have to type your passphrase for the private key)

docker run -v $OVPN_DATA:/etc/openvpn --rm -it kylemanna/openvpn ovpn_initpki

Run the VPN service: start and detach the container (-d) and map a host port to the UDP container port where the openvpn server process is listening (1194). In this example the host port will be 1195

docker run -v $OVPN_DATA:/etc/openvpn -d -p 1195:1194/udp --cap-add=NET_ADMIN kylemanna/openvpn

Generate client configuration (i.e., add a user to the VPN). If you omit the nopass option, the client key will be encrypted with a passphrase.

docker run -v $OVPN_DATA:/etc/openvpn --rm -it kylemanna/openvpn easyrsa build-client-full vince nopass

The client key will be in ${OVPN_DATA}/pki/private and the certificate in ${OVPN_DATA}/pki/issued
Retrieve the client configuration to a local file:

docker run -v $OVPN_DATA:/etc/openvpn --rm kylemanna/openvpn ovpn_getclient vince > vince.ovpn

If you need to add more users, just repeat the last two steps to create a user configuration on the server and retrieve the ovpn file.

Full reference: https://thealarmclocksixam.com/2017/03/07/setup-your-vpn-in-docker-with-openvpn-in-5-minutes/


How to handle availability zones and failover with ansible? Is it even within the scope of ansible?

What your are describing can typically be addressed with a correct inventory design.

[zone1]
serverA
serverB
serverC

[zone2]
server1
server2
server3

[zone3]
serverX
serverY
serverZ

[jenkins_master]
serverA
server1
serverX

[jenkins_slave]
serverB
serverC
server2
server3
serverY
serverZ

From there you can easily:

create plays that will target a group or a pattern and that you can limit to an other group or pattern at runtime, e.g. target jenkins_master and run the playbook on all of them or limit it to zone1.
create inventory vars for your hosts and groups so that e.g. servers in zone1 will know what is the uri of the master in that zone.

This is a super-quick overlook. I suggest you read in depth working with inventory to get a better understanding of all the inventory concepts and pay attention to the part concerning the organization of variables in the inventory for groups and hosts


While Romeo has laid out the basic facts I'd like to add a bit of experience to the answer:

Basics Cacti is focused on graphs.  It doesn't do any of the up/down monitoring that is provided by something like Nagios.  Classically Nagios does not provide any useful graphs, but more on that later.
Config Scaling Cacti tends to be configured through the GUI manually.  Eventually you will hopefully scale enough to find this frustrating.  I've switched to Munin in places where there's lots of configuration management and it is easy to turn a server inventory into a config file with templates.  Nagios tends to be configured with config files so hopefully you script the translation of your inventory in that format too.  Then you can scale to thousands of nodes without redesigning your monitoring system.
Distinctions blur. The classic distinction between Cacti/Nagios of graphing vs up-down status has been blurred in a number of Nagios derivatives.  Icinga and OMD, for instance, integrate graphing into a Nagios-style monitoring architecture.
Plugin Availability Doing something in the Nagios world gives you access to tons of well tested plugins to monitor almost anything under the sun.  Cacti also has plugins, but not nearly as many.

So obviously my Cacti experience hasn't been that great and I'd recommend you look elsewhere for your long-term graphing needs.


Accessing attributes from within recipes

I think that according to chef documentation you should use normal with node data.

  A normal attribute is a setting that persists in the node object. A
  normal attribute has a higher attribute precedence than a default
  attribute.

you can read about it here:

https://docs.chef.io/attributes.html

https://docs.chef.io/knife_node.html

some example:

  "normal": {
    "ebs": {
      "devices": [
        {
          "mount_device": "/dev/xvdf",
          "encrypt": true,
          "volume_mapping": [
            {
              "device": "/dev/xvdf"
            }
          ]
        }
      ]
    },

In you recipe:

if node.ebs.devices[0]['mount_device'] == '/dev/xvdf' and node.ebs.devices[0]['encrypt'] ...

Another example with node['myattribute']

   "normal": {
        "myattribute": "myvalue"
        },


How to shut down a stateless app gracefully?

kill -9 is used by the kernel to terminate stateful programs that are no longer responding (including to shutdown requests). To gracefully shut down a stateful app you would use kill -15 which instructs the application to call it's shutdown functions and do the requisite cleanup. If you app is truly stateless, there should be no difference in the results of kill -9 and kill -15, but technically, a graceful shutdown of this program would still be kill -15


Ansible playbook for configuring runit to supervise an uberjar?

So both have their place. On one hand, putting service definitions in the package helps to keep the number of "moving pieces" to a minimum. On the other, it also requires a more complex package build process to at least some degree (like you would have to build your own packages for things instead of using distro packages). If you have the capability to do it in packages, that seems better since moving pieces in prod will always eventually break.


The error shows that you are using AWS ECR credentials to login to dockerhub which will not work. Dockerhub is a different registry(both public and private), however ECR is private registry only.

For pulling public images from dockerhub there is no need to login to dockerhub. You can simply use docker pull command and it will pull an image from dockerhub registry.

Example: docker pull mongo

However, when you want to pull an image from ECR, you need to first login to the AWS ECR and then only you can pull an image from ECR.


how long does it take to deploy a serverless function?

This will vary based on your cloud provider and what services you are using.  AWS Lambda functions are designed to be highly available for the region.  Per the AWS Lambda FAQ:

  When you update a Lambda function, there will be a brief window of time, typically less than a minute, when requests could be served by either the old or the new version of your function.

So the new function will not be instantaneously available, and there is a possibility that the old function may be called.


What does 1 vcpu mean precisely/exactly?

vcpu = Virtual CPU, that's just the number of core of the instance. Nothing more, nothing less.

For the rest of your questions they're unrelated, this depends on the underlying hypervisor's physical processors, there's no guarantee they use the same type with same frequency.


How to get memory usage per process with sar, sysstat?

As Tensibai mentioned, you can extract this info from the /proc filesystem, but in most cases you need to determine the trending yourself. There are several places which could be of interest:

/proc/[pid]/statm

          Provides information about memory usage, measured in pages.
          The columns are:

              size       (1) total program size
                         (same as VmSize in /proc/[pid]/status)
              resident   (2) resident set size
                         (same as VmRSS in /proc/[pid]/status)
              shared     (3) number of resident shared pages (i.e., backed by a file)
                         (same as RssFile+RssShmem in /proc/[pid]/status)
              text       (4) text (code)
              lib        (5) library (unused since Linux 2.6; always 0)
              data       (6) data + stack
              dt         (7) dirty pages (unused since Linux 2.6; always 0)

cat /proc/31520/statm
1217567 835883 84912 29 0 955887 0

memory-related fields in /proc/[pid]/status (notably Vm* and Rss*), might be preferable if you also collect other info from this file

          * VmPeak: Peak virtual memory size.

          * VmSize: Virtual memory size.

          * VmLck: Locked memory size (see mlock(3)).

          * VmPin: Pinned memory size (since Linux 3.2).  These are
            pages that can't be moved because something needs to
            directly access physical memory.

          * VmHWM: Peak resident set size ("high water mark").

          * VmRSS: Resident set size.  Note that the value here is the
            sum of RssAnon, RssFile, and RssShmem.

          * RssAnon: Size of resident anonymous memory.  (since Linux
            4.5).

          * RssFile: Size of resident file mappings.  (since Linux 4.5).

          * RssShmem: Size of resident shared memory (includes System V
            shared memory, mappings from tmpfs(5), and shared anonymous
            mappings).  (since Linux 4.5).

          * VmData, VmStk, VmExe: Size of data, stack, and text
            segments.

          * VmLib: Shared library code size.

          * VmPTE: Page table entries size (since Linux 2.6.10).

          * VmPMD: Size of second-level page tables (since Linux 4.0).

          * VmSwap: Swapped-out virtual memory size by anonymous private
            pages; shmem swap usage is not included (since Linux
            2.6.34).

server:/> egrep '^(Vm|Rss)' /proc/31520/status
VmPeak:  6315376 kB
VmSize:  4870332 kB
VmLck:         0 kB
VmPin:         0 kB
VmHWM:   5009608 kB
VmRSS:   3344300 kB
VmData:  3822572 kB
VmStk:      1040 kB
VmExe:       116 kB
VmLib:    146736 kB
VmPTE:      8952 kB
VmSwap:        0 kB

Some processes can, through their behaviour and not through their actual memory footprint, contribute to the overall system memory starvation and eventual demise. So it might also be of interest to look at the OOM Killer related information, which already takes into account some trending information:

/proc/[pid]/oom_score

          This file displays the current score that the kernel gives to
          this process for the purpose of selecting a process for the
          OOM-killer.  A higher score means that the process is more
          likely to be selected by the OOM-killer.  The basis for this
          score is the amount of memory used by the process, with
          increases (+) or decreases (-) for factors including:

          * whether the process creates a lot of children using fork(2)
            (+);

          * whether the process has been running a long time, or has
            used a lot of CPU time (-);

          * whether the process has a low nice value (i.e., > 0) (+);

          * whether the process is privileged (-); and

          * whether the process is making direct hardware access (-).

          The oom_score also reflects the adjustment specified by the
          oom_score_adj or oom_adj setting for the process.

server:/> cat proc/31520/oom_score
103

/proc/[pid]/oom_score_adj (or its deprecated predecessor /proc/[pid]/oom_adj, if need be)

          This file can be used to adjust the badness heuristic used to
          select which process gets killed in out-of-memory conditions.

          The badness heuristic assigns a value to each candidate task
          ranging from 0 (never kill) to 1000 (always kill) to determine
          which process is targeted.  The units are roughly a proportion
          along that range of allowed memory the process may allocate
          from, based on an estimation of its current memory and swap
          use.  For example, if a task is using all allowed memory, its
          badness score will be 1000.  If it is using half of its
          allowed memory, its score will be 500.

          There is an additional factor included in the badness score:
          root processes are given 3% extra memory over other tasks.

          The amount of "allowed" memory depends on the context in which
          the OOM-killer was called.  If it is due to the memory
          assigned to the allocating task's cpuset being exhausted, the
          allowed memory represents the set of mems assigned to that
          cpuset (see cpuset(7)).  If it is due to a mempolicy's node(s)
          being exhausted, the allowed memory represents the set of
          mempolicy nodes.  If it is due to a memory limit (or swap
          limit) being reached, the allowed memory is that configured
          limit.  Finally, if it is due to the entire system being out
          of memory, the allowed memory represents all allocatable
          resources.

          The value of oom_score_adj is added to the badness score
          before it is used to determine which task to kill.  Acceptable
          values range from -1000 (OOM_SCORE_ADJ_MIN) to +1000
          (OOM_SCORE_ADJ_MAX).  This allows user space to control the
          preference for OOM-killing, ranging from always preferring a
          certain task or completely disabling it from OOM killing.  The
          lowest possible value, -1000, is equivalent to disabling OOM-
          killing entirely for that task, since it will always report a
          badness score of 0.

          Consequently, it is very simple for user space to define the
          amount of memory to consider for each task.  Setting an
          oom_score_adj value of +500, for example, is roughly
          equivalent to allowing the remainder of tasks sharing the same
          system, cpuset, mempolicy, or memory controller resources to
          use at least 50% more memory.  A value of -500, on the other
          hand, would be roughly equivalent to discounting 50% of the
          task's allowed memory from being considered as scoring against
          the task.

          For backward compatibility with previous kernels,
          /proc/[pid]/oom_adj can still be used to tune the badness
          score.  Its value is scaled linearly with oom_score_adj.

          Writing to /proc/[pid]/oom_score_adj or /proc/[pid]/oom_adj
          will change the other with its scaled value.

server:/> cat proc/31520/oom_score_adj
0


AWS Security Group Creation Using AWS CLI

Ok I got the answer after exploring few things and looking at the error more attentively. Here is the way:

aws ec2 authorize-security-group-ingress --group-id sg-07xxxx24d  --protocol tcp --port 22 --source-group sg-d7dyyyy5


How to create GitHub Actions for unit testing in .NET projects?

You do not need to include anything into your assembly to run your tests using GitHub Actions. Just create workflow file in .github/workflows folder with the following content (assuming that you have .NET Core project):

---
name: Tests

on: push

jobs:
  tests:
    name: Unit Testing
    runs-on: windows-latest
    steps:
      - uses: actions/checkout@v2.1.0
      - run: dotnet test

  dotnet is pre-installed on windows machine but is not pre-installed on macos and ubuntu. So, you have to install dotnet by adding an extra step in case you want to run it on one of these machines. You can use actions/setup-dotnet action for this purpose.


Jenkins Pipeline "Build when a change is pushed to GitLab"

Looks like you are using the Gitlab plugin for jenkins?
if so this link should be of help https://github.com/jenkinsci/gitlab-plugin#declarative-pipeline-jobs


Is there any way I can rotate AWS KMS keys every time interval < 1 year?

The AWS documentation is clear that automated key rotation is only supported by Amazon on a yearly basis. However, you can use the CLI to manually create new keys, and then use an automation technique such as cron to automate the manual process.

In this case you can likely have cron run a bash script to read from a list of keys and credentials and then use the AWS kms cli to "manually" change your keys.


Unable to connect to Elasticsearch [ Error: Request Timeout after 30000ms ]

Put the following setting in /etc/elasticsearch.yml and it will work.
network.host: 0.0.0.0
discovery.type: single-node


Based on the github reply I received, it looks only way is to allow it in Jira as it will spam same alert otherwise:

  Labels are how JIRAlert matches alerts to tickets. Without them, it
  would simply create a new ticket every time it gets called by
  Alertmanager for the same alert, i.e. once every few minutes.

  If I understand the error correctly, it looks like you'll have to
  enable labels for your JIRA project. I don't have admin access to a
  JIRA instance, so I can't tell you exactly how to do that.


Understanding and tweaking AWS policies for job functions

Sure you can have a look at the JSON policy documents. On that page that you link to (AWS Managed Policies for Job Functions) click the Policy you're interested in while being logged to your AWS account and it will take you to the policy definition.

E.g. if you click to AdministratorAccess it will open the IAM Policies page and show you the JSON:

Hope that helps :)


How many DB instances to use? When to switch to container services?

The choice of the approach is a risk assessment, if one of the DB client (application) goes wild and bring your server instance down, what is the cost in term of human hours lost during the outage ?
If you run the DB yourself instead of managed you have to include your time costs to update it, backup it, monitoring it, etc. is it really cheaper ?
All this is very company specific and need to take into account the human cost, does a staging environment down block 1 or 80 devs ? Are the devs junior or senior ? The cost per hour of outage change a lot from these simple two metrics.
At the end of the day, that's a judgement on accepted risk of outage and capacity to restart versus administration time to avoid the outage for which there's no magical mathematics formula.


How do I make read only mount in Docker container writable?

Solution to this problem is to create an overlay filesytem over a read-only mount, but if you try to do it directly, overlay will refuse to put upper and work directories on another overlay filesystem. The trick is to create a tmpfs for the upper and work directories like this:

Create a script called run-in-c.sh

#!/bin/bash
NAME=$1
shift
HOSTNAME=dock${NAME}
CONTAINER=dc-${USER}-${NAME}
REPOSITORY=${HOME}/repository
BASEIMAGE=hub.docker.io/my-org/my-base-container
OVERLAY=/mnt/overlay
LOWERDIR=/mnt/lower
UPPERDIR=${OVERLAY}/upper
WORKDIR=${OVERLAY}/work
TARGET=/mnt/repository
# PRIVILEGED="--privileged"
PRIVILEGED="--cap-add SYS_ADMIN"
docker container create --name $CONTAINER $PRIVILEGED --hostname $HOSTNAME --volume ${CO}:${LOWERDIR}:ro $BASEIMAGE
docker container start $CONTAINER
docker container exec $CONTAINER mkdir -p $OVERLAY
docker container exec $CONTAINER mount -t tmpfs tmpfs $OVERLAY
docker container exec $CONTAINER mkdir -p $WORKDIR $UPPERDIR $TARGET
docker container exec $CONTAINER mount -t overlay overlay -o lowerdir=${LOWERDIR},upperdir=${UPPERDIR},workdir=${WORKDIR} $TARGET
docker container exec -it --workdir $TARGET $CONTAINER $*
docker container stop $CONTAINER
docker container rm $CONTAINER

Then you can run commands in the container as:

run-in-c.sh test01 'cd dir && command args'

or simply get interactive shell with:

run-in-c.sh naming-stuff-is-too-hard bash


What is happening in an Azure Devops build and release?

Lets take this line by line:

My additions are added as comments.

trigger:
- master
# The branch you are building.

pool:
  vmImage: 'Ubuntu-16.04'
# The type of VM your code will run on.

variables:
  buildConfiguration: 'Release'
#This ends up defining which webconfig file to use.

steps:
- script: dotnet restore
  displayName: Restoring Dependencies

- script: dotnet build --configuration $(buildConfiguration)
  displayName: 'dotnet build $(buildConfiguration)'
# Builds your application
- task: DotNetCoreCLI@2
  displayName: Unit Tests
  inputs:
    command: test
    projects: '**/*Tests/*.csproj'
    arguments: '--configuration $(buildConfiguration)'
#Runs unit tests for your application
- task: DotNetCoreCLI@2
  displayName: Publishing
  inputs:
    command: publish
    publishWebProjects: True
    arguments: '--configuration $(BuildConfiguration) --output $(Build.ArtifactStagingDirectory)'
    zipAfterPublish: True
#Pushes your application as an artifact (You'll be able to see it in your list of artifacts in Azure DevOps)
- task: PublishBuildArtifacts@1

As for your release script, you are forgetting to pass in the values so your release script isn't doing anything at the moment.
But when it does work, it will take the objects from your artifacts or container images and deploy them into your webapp the same way it would do it if you ran those commands from your local machine.


Terraform: How to support releasing new versions of the code, running in parallel with the last stable release?

There are a few different ways to achieve goals of this sort, each with some different tradeoffs. I'm going to describe the most common ones below.

The simplest approach is to use Terraform's create_before_destroy mechanism with autoscaling groups. An example of this pattern is included in the aws_launch_configuration documentation.

In this scenario, changing the AMI id causes the launch configuration to be re-created. Due to create_before_destroy, the new configuration is created first, then a new autoscaling group is created, adding the new instances to an attached ELB. The min_elb_capacity argument to aws_autoscaling_group can be used to ensure that a given number of instances are present and healthy in the attached ELB before considering the autoscaling group to be created, thus delaying the destruction of the old autoscaling group and launch configuration until the new one is serving requests.

The downside of this approach is the lack of control it represents. Since Terraform is thinking of the entire set of changes as a single run, it's impossible to pause after creating the new instances to allow other checks to be carried out before destroying the old ones. As a consequence, the ELB healthcheck is the only input to deciding if the new release is "good", and rolling back is impossible once the old resources have been destroyed.

A second common approach is to adopt a sort of "blue/green deployment" pattern with explicit changes to two clusters. This is done by putting all of the per-release resources in a child module, and instantiating that module twice with different arguments. In the top-level module this would look something like the following:

resource "aws_elb" "example" {
  instances = "${concat(module.blue.ec2_instance_ids, module.green.ec2_instance_ids)}"

  # ...
}

module "blue" {
  source = "./app"

  ami_id = "ami-1234"
  count  = 10
}

module "green" {
  source = "./app"

  ami_id = "ami-5678"
  count  = 0
}

The principle of operation here is that in the "steady state" (no deployment in progress) only one of these modules has a non-zero count, and the other one has zero. During a deployment, they are both set to the same non-zero count, but with different ami_id values. Each deployment swaps which of the modules is the "active" module, with both being active during the deployment.

When using this approach, each step is a distinct Terraform operation:

change count of the inactive module to nonzero and set its AMI id
apply the change with Terraform, thus activating the new module
verify that the new release is good
change count of the older module to zero
apply the change with Terraform, thus deactivating the old module

Although this has more steps, it allows arbitrary verification and an arbitrary amount of time to pass during step 3. It also allows "rolling back" by resetting the previously-inactive cluster count to zero.

Since both the old and new clusters exist in the same configuration, there is the risk of using this pattern incorrectly and prematurely destroying the active cluster. This can be mitigated by carefully reviewing Terraform's plan to make sure it leaves the old cluster untouched, but Terraform itself can't guarantee this.

Also, since both clusters are using the same child module configuration, it can be tricky to make updates to that configuration while retaining the blue/green separation. If changes are made that would require Terraform to replace the running instances, it's necessary to temporarily have two copies of the module code on disk, make the source arguments point to separate copies, and make the change only to the copy used by the inactive module.

The final approach I'll present is the most extreme and manual, but it does the best job of meeting your requirements and retaining control. This is, in effect, the most literal interpretation of your current CloudFormation workflow, and is a more concrete version of the approach you talked about in your question.

In this approach, there are two entirely-separate Terraform configurations, which I will call "version-agnostic" (things that must survive between versions, such as your ELB) and "version-specific" (the resources that are re-created for each new version).

The version-agnostic configuration will contain the ELB and will, as you suspected, export its id for consumption by the version-specific configuration:

terraform {
  required_version = ">= 0.9.4"
  backend "s3" {
    bucket = "example-company-terraform-state"
    key    = "exampleapp/version-agnostic"
    region = "eu-central-1"
  }
}

resource "aws_elb" "example" {
  # ...
}

output "elb_id" {
  value = "${aws_elb.example.id}"
}

This configuration can be initialized, planned and applied as usual, creating an ELB with no attached instances to start.

The version-specific configuration would be similar to the "app" child module in the previous approach, but this time as a top-level module. The backend configuration for this module would omit the S3 key, since this will change for each new release as you expected:

terraform {
  required_version = ">= 0.9.4"
  backend "s3" {
    bucket = "example-company-terraform-state"
    region = "eu-central-1"
  }
}

The specific key can then be set (or re-set) when running terraform init:

$ terraform init -reconfigure -backend-config="key=exampleapp/20170808-1"

Here I chose to use a "current date, release index" tuple as an identifier for a release. By running terraform init with a new value for this argument, an entirely separate state is created, independent of the last. Using -reconfigure tells Terraform that you don't wish to migrate the old state to the new, but rather to just switch directly to the new state path, possibly creating a new state in the process.

You can then run terraform show to confirm that indeed the state is empty (and thus operations won't affect existing resources) and then run a plan/apply cycle as normal.

Once you're satisfied with the new release, you can switch back to the previous version and destroy it.

The version-specific configuration will need the id of the ELB from the version-agnostic configuration in order to populate the load_balancers attribute of aws_autoscaling_group. To get access to this, we can use the terraform_remote_state data source to read the values from its state in S3:

data "terraform_remote_state" "version_agnostic" {
  backend = "s3"
  config {
    bucket = "example-company-terraform-state"
    key    = "exampleapp/version-agnostic"
    region = "eu-central-1"
  }
}

resource "aws_autoscaling_group" "example" {
  # ...

  load_balancers = ["${data.terraform_remote_state.version_agnostic.elb_id}"]
}

With a system of this complexity, it would likely be best to run Terraform via some sort of wrapper script or orchestration to make the release process less onerous. For example, such a script might automatically generate the new version number to avoid the risk of a human operator mistyping the date or accidentally conflicting with an existing one. There's some recommendations and caveats about running Terraform via scripts in the guide Running Terraform in Automation.

Although the third option here is the most direct mapping of your CloudFormation approach, the second option is more commonly used due to it striking a reasonable compromise between control and workflow overhead.


Dockerized Flask: Connection reset by peer

Trying to connect from outside fails

Are you actually connecting from outside? Flask is binding to localhost (127.0.0.1) and that will only be reachable from within the container. If you're on your local machine, you'll need flask to bind to all IP's:

from flask import Flask
app = Flask(__name__)

@app.route('/')
def hello():
    return "Hello World!"

if __name__ == "__main__":
    app.run(host='0.0.0.0')

That works for me successfully.


You have not instantiated the Utils class in the myorg classpath so you cannot reference the method echoTest(). Try Utils.echoTest or just echoTest()


Specifying branch name for jenkins job through AnyStatus

Oh, I figured it out. I just went to the job configuration, activated the "build is parameterized" checkbox, added a string parameter called BRANCH, and in the "branches to build" section added ${BRANCH}. Pretty much what is explained here:

https://stackoverflow.com/questions/20359810/how-to-trigger-jenkins-builds-remotely-and-to-pass-parameters
https://stackoverflow.com/questions/32108380/jenkins-how-to-build-a-specific-branch


How to get the ARN for an arbitrary Amazon Resource?

https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html lists the schema for (I believe) all of the ARNs that Amazon supports.  Text in italic red indicates where you should substitute in your own value, rather than the literal string, and there are examples in each section to help figure this out.

Be warned that IAM is less usefully restrictive than it appears on first glance.  To start with, most services have only partial IAM support.  Secondly, often the things that are supported at a resource level are supported only in very specific ways; for instance, granting all permissions on a subnet ARN does not grant the attached users the ability to fully manage instances in that subnet, as AttachVolume only supports restricting by volume id, etc.  Thirdly, if the users are going to use the Amazon web console at all, they must have broad viewing permissions, as the console was not written with PoLP in mind and issues non-resource-restricted API calls (and often merely dumps errors in the javascript console!).


How to automate rolling upgrades for ec2 running linux behind elastic and/or application load balancer?

AWS have a specific service to deal with this type of use case, EC2 Systems Manager

This allows making base patch sets and then checking they are applied to target systems.

  "Patch Manager is a fully-managed service (provided at no additional
  cost) that helps you simplify your operating system patching process,
  including defining the patches you want to approve for deployment, the
  method of patch deployment, the timing for patch roll-outs, and
  determining patch compliance status across your entire fleet of
  instances. It’s extremely configurable with some sensible defaults and
  helps you easily deal with patching heterogeneous clusters."

-- "Amazon EC2 Systems Manager Patch Manager now supports Linux"

You'll need to install an SSM agent on each system and add a role but other than that it should be unobtrusive


Sub-subdomain Domain name for different environment

Both are completely correct. It can differ organisation by organisation.
Also, once you start using one of the domain namespace type, you will gradually remember the format and no issues will come. However, I can see one issue here which is related to SSL.

If you go with any one of the above mentioned namespaces, you need to generate multiple SSL certificates such as *.api.superproduct.com and *.front-end.superproduct.com.

If you go with other namespace type then SSL certs will be required for *.staging.superproduct.com, *.qa.superproduct.com and *.development.superproduct.com.

To avoid this, I would suggest you to have the domains in either of the following formats.

<component-name>-<env-name>.superproduct.com

Examples are:
api-qa.superproduct.com, api-staging.superproduct.com etc.

<env-name>-<component-name>.superproduct.com

Examples are:
qa-api.superproduct.com, staging-api.superproduct.com etc.

In this case, only one SSL certificate for *.superproduct.com will be required.

You can go ahead with the best possible option as per your requirements and analysis.