Pagerduty API integration and auto-resolution

PD has a lot of native integrations that can do auto-resolve but for those that don't you can use the API and/or rulesets to do that.
Here is a link to the ruleset documentation: https://support.pagerduty.com/docs/rulesets
And here is documentation on the incident resolution API:
https://developer.pagerduty.com/api-reference/reference/REST/openapiv3.json/paths/~1incidents~1%7Bid%7D/put
In the API call you would pass the following json body to resolve it:
{
"incident": {
"type": "incident_reference",
"status": "resolved"
}
}


Handle OS and Software maintenance/updates on Hardware distributed to Customers

I can provide some personal experience here. I worked at a company that had the same need. The product was a BeagleBone (like an Arduino or Raspberry Pi) that ran our software and sent data back to our SaaS. It had to be installed inside customer networks, and be a little black box that they didn't touch.

When I walked into this company, management of the systems was a nightmare. Networking was a nightmare. OS updates, keeping them online, shipping new versions of the software, etc... it was a nightmare.

The path we ended up choosing ended up working really well: The entire application and supporting packages was shipped as a Docker container. That allowed us to easily ship application code changes and control versions of any supporting software (ffmpg, gcc, etc...).

With the application abstracted, that simply left making the OS as minimal as possible. All we needed was a dead simple OS that could run docker. One of our low level engineers actually made our own in-house fork of Debian (if I remember) and we shipped that embedded directly into the devices.

We already had a service contract with all of our customers to service these machines yearly, so when our field technician went on-site, the procedure was to replace the entire unit with a fresh hardware that came with the latest embedded OS. We didn't have to make many OS changes just to run Docker, so often times swapping wasn't even necessary. But out of procedure, we always swapped hardware whether it was an emergency service call or the yearly maintenance.

I can't say if this would be a standard approach or not, but just that it worked really well for us and was a life saver. It sounds like your idea of shipping the OS as an immutable image is along the same lines.

Edit: When I left, we were still using home grown bash scripts to automate the Docker container deployments, but the idea was to move toward a config management tool (Like Chef, Salt, or Ansible). The basic premise is that you have a private Docker registry somewhere, then on the devices you would simply run a couple commands:

docker pull my-private-registry/my-custom-image:latest
docker run my-custom-image:latest

If you are going to always be using the 'latest' tag, you could then have your devices ship with a cronjob that has then always shut down their local running Docker container, and re-pull the 'latest' tag every week during a scheduled maintenance window.


You can update the daemon.json config to specify your normal syslog driver (assuming linux) to use the normal syslog logrotation method for keeping log files or shipping off-box
https://docs.docker.com/config/containers/logging/syslog/
{
  "log-driver": "syslog",
  "log-opts": {
    "syslog-address": "udp://localhost:514"
  }
}


How do I release multiple projects to same domain using AWS when developer commit the latest code change

You may wish to try scripting the actions you want. Then use git hooks to trigger your scripts to release the latest code to your environment. Alternatively, you could use something like Jenkins to configure a "push on green" system if you are working with compiled code. Lastly, (though I haven't used them) Amazon claims that their CodePipeline product can do this (and perhaps CodeBuild can as well), though my personal preference is to keep things as generic as possible and avoid drinking Amazon's Kool-Aid in order to keep things portable. Like a drug dealer, the first hit may be free with Amazon, and things are great until they have you hooked and decide to jack up the price. Utilizing my own open source system ensures that I can easily take my ball and go to another cloud or home altogether (and host it myself) in the event that I need to make a rapid exit from Amazon's cloud and avoids lock-in.


How to Configure jenkins email notification for some commercial email services?

You may consider these two Jenkins plugins for it:

https://wiki.jenkins.io/display/JENKINS/Mailer

https://wiki.jenkins.io/display/JENKINS/Email-ext+plugin

The first one is mostly basic and the second one is extended for additional user customizations. All necessary instructions regarding configuration, etc are described there.

      How we can get mail server setting details like mail server name, port number, using SSL, etc. for such email services?

I am afraid that the best approach here is either refer to your system administrator who is resposible for it or contact with the provider who provides mail services.


Verify that you have the correct CMake Plugin installed. Also, add parenthesis to your cmakeBuild statement in your pipeline

pipeline {
  agent any
  stages {
    stage('build') {
      steps {
          cmakeBuild(
            installation: 'InSearchPath'
          )
      }
    }
  }
}


How to kill all Docker containers based on a given image?

This will do want I want: killing all containers directly based on mariadb regardless of their tag--and without killing containers having mariadb somewhere higher in their ancestor list.

sudo docker kill $(
  sudo docker ps --format '{{.ID}} {{.Image}}' | awk '$2 ~ /^mariadb:/ { print $1}'
)


Are there any best practices and/or tools for 'nohup' monitoring?

I'm not sure it's a best practice but what I would do is using filebeat to read the log in near real time and push it to ElasticSearch.
AWS provides an ElasticSearch Service which can save you from this part.

The drawback of using filebeat directly to elasticsearch is that the time of events will be the time the prospector read them and if your logs have a timestamp they can be "ingested" in different order than produced (from multiple workers).

You can add Logstash inbetween, sending from filebeat to logstash and then to elasticsearch. Logstash is able to interpret your logs and split them into fields to search again, in the same manner you can parse your date in log so events will be stored with a correct timestamp.

You may use logstash directly to read the logs, but it's quite heavy on resources so I would keep it separated to no impact the workers.

If you buy a license from x-pack you can add alerts on messages directly from elasticsearch.

If you want to save you from the burden of maintaining logstash there's services like logmatic.io which provides you with the full stack, you just have to push your logs and a bunch of parsing are handled from stock.


Service account permissions error when provisioning instance group from template

Try adding permissions on your project level:

gcloud projects add-iam-policy-binding <your-project-id> --member=serviceAccount:terraform@vpc-challenge-lab-245114.iam.gserviceaccount.com --role='roles/iam.serviceAccountUser'


Terraform: How to use a variable inside of another variable?

Largely inspired by the documentation the idea is to create a terraform datasource from a filter of the available resources and then use this as entry point in your ami resource:

variable "security_group_id" {}
variable "subnet_id" {}

data "aws_security_group" "selected" {
  filter {
    name = "${var.ansible_ami["security_groups_name"]}}"
  }
}

data "aws_subnet" "selected" {
  filter {
    name = "${var.ansible_ami["subnet_name"]}"
  }
}

resource "aws_instance" "ansible" {
    ami = "${var.ansible_ami["ami"]}"
    instance_type = "${var.ansible_ami["size"]}"
    security_groups = "${data.aws_security_group.selected.id}"
    subnet_id = "${data.aws_subnet.selected.id}"
    associate_public_ip_address = "${var.ansible_ami["enable_public_ip"]}"
    #user_data = ""

I'm not using terraform myself, so this could be plain wrong.


How to allow users to list authorized buckets in GCP Storage?

It's not currently possible to list all the accessible buckets across all cloud projects, neither in the developer console browser nor via the gsutil command. Both of them only display buckets for one project at a time.

But if you specify a particular project (or stick with the one selected by default) you can list the accessible buckets inside that project in both gsutil (as shown in @chupasaurus's answer) and in the developer console browser. In the browser you get to the project selection popup by clicking on the project's dropdown menu in the top navigation bar.

You could try to file a feature request for it at https://issuetracker.google.com/components/187243/manage#basic, it might get accepted if it shows enough interest.


Can the Angular CLI be used to build and deploy on a remote server via ssh?

You need to use continuous deployment tools.
Not available yet.


EKS - created cluster from console with federated IAM admin - how to access

The only way to manage IAM users - using appropriate account with IAM Administrator privileges.

See AWS Identity and Access Management
If you lost your root account, you may contact an AWS support, see Recover Your AWS Account Password


Ok, solved it :)

Here the task for everyone who intends to do the same.

Only the skip/changed console output could be replaced by a nicer message.

- name: Register unwanted motd files
  stat:
    path: "{{motd_path}}/{{item}}"
  register: filecheck
  with_items: "{{ unwanted_motd_files | default([]) }}"

- name: Check if unwanted motd files are executable and remove the executable bit
  file:
    path: "{{motd_path}}/{{item.item}}"
    state: touch
    mode: u-x,g-x,o-x
  with_items: "{{ filecheck.results }}"
  when: item.stat.exists == true and item.stat.executable == True


For your first question:

  When using a Kubernetes LoadBalancer or Ingress, is the normal
  NodePort used as the underlying connection point?

Technically it depends on the cloud network for LoadBalancers (e.g. Google Compute Engine does not technically require this although one may be created).  However, I believe a NodePort and ClusterIP are generally created.

For your question on the usage of NodePorts:

The short answer is yes, they are safe to use if you know what you are doing.

The longer answer is that NodePorts are great for debugging and development but have some issues in production:

You can only have one service on a port.
HTTP/HTTPS traffic may be routed through a non-standard port.
The port will be opened on all nodes.

However, NodePorts do provide some nice flexibility (from Kubernetes documentation):

  Using a NodePort gives you the freedom to set up your own load balancing solution, to configure environments that are not fully supported by Kubernetes, or even to just expose one or more nodes’ IPs directly.


What are the advantages of dockerizing nginx and php in different containers?

PHP with nginx is usually done using php-fpm which is a separate processus.

Keeping the core idea of docker of one process (see end of answer for more details on this point) per container this makes sense to have the nginx process and php-fpm process in separate containers.

As the communication between nginx and php-fpm arise through fastcgi the php-fpm container can also be on a separated host and this allows using a cluster of php-fpm containers behind nginx.

After the wall of comment here's a little more background, docker documentation have paragraph about the idea that a container should have only one concern.

The main idea of a linux container (lxc) is to run a process in an isolated namespace at the cpu and memory level, docker add on top of this an isolation at the filesystem level.
The advantage is that the compromission of a process within this namespace won't allow to read memory of other processes and as such should prevent other compromission on the host.

While talking about nginx and php-fpm, they work in pair but each has it's own concern, nginx will do the HTTP part, routing, headers validation, etc. and php-fpm will do the code interpretation and return the html part to nginx. While it's usual to have both together serving a single application that's not mandatory.

Depending on context it may be easier to have a container including the whole stack for an application, on a developer workstation for exemple. But ideally for production use, try to keep the fewer interaction inside the container, having separated processes in the same container with supervisord brings its share of problem in term of zombie process and log handling (exemple story here for illustration purpose only).

So finally I'll quote the docker page with some emphasis:

  While “one process per container” is frequently a good rule of thumb, it is not a hard and fast rule. Use your best judgment to keep containers as clean and modular as possible.

There's no "silver bullet rule" which apply to everything, it's always a balance between the complexity within the container and the complexity orchestrating the containers themselves.


Impact on policy after deleting IAM user

The policy will still exist.

The documentation lists what is deleted when a user is deleted, any resources that the user has created will persist.


Reuse of docker swarm containers for batch jobs

Correct me if I am wrong, but that is one of the reasons why docker orchestration tools exists like docker-swarm to be able to deploy multiple docker containers by using one command line.

  Create a service with 5 replica tasks (–replicas)

  Use the --replicas flag to set the number of replica tasks for a replicated service. The following command creates a redis service with
  5 replica tasks:

$ docker service create --name redis --replicas=5 redis:3.0.6

4cdgfyky7ozwh3htjfw0d12qv

If for example 1000 identical containers that use a different environment variable have to be deployed, multiple docker service create could be added to a script, instead of copy-paste multiple times in one docker-compose file.


Your main problem is that you're flushing the CDN when you're deploying a new set of artefacts, which will effectively ruin anything still in progress (whatever its state). You should add the new artefact versions and leave the older ones in place, at least for as long as your published cache lifespan for the top index.html file is.

This way even if users look at an older (cached) version of the page which references other older version artifacts they'll still be able to download them (users have no clue that a new version is available as the cached version is immediately served as long as it's not expired). And whenever the cache expires users will get the new version of the top page, with the references to the new artefact versions.


Using Red Hat Docker image as base image

As long as you have access to the FROM image registry, you are totally free to choose where to pull from and where to push to.


Running Automation on Google Cloud Virtual Instances using Ansible

You run the command with the user shishir9159_gmail_com but the error shows shishir9159@35.202.219.6: Permission denied ... seems like the _gmail_com portion of the username is truncated. Can you try setting the ansible_user config parameter in your inventory file or playbook rather than passing it on via the CLI?


Executing a command in running docker swarm service

I suggest making a cronjob container that does the SQL dump and elastic index using docker exec commands toward the corresponding containers.


How to use feature toggles and frequent deployment under SOX compliance?

It depends on how you're managing your feature flags. Ideally, you're using a flagging system that allows you to target specific users or segments of users so that an untested feature isn't inadvertently enabled for all your users. Some feature management systems (like LaunchDarkly) allow you to target users any way you want.

For example, a common workflow (for teams who use feature flags) is to release a new feature under a flag that's only turned on for the product team that's working on that feature. So, even if the feature hasn't been tested, it's guaranteed that only the right users have access to that new feature.

This allows your product team to continue to work on the feature and test it in production (if that's appropriate). Of course, a full set of tests should cover that feature before it's released, but it's nice that product teams have a way to iterate quickly without the hangups of a process that didn't involve feature flags.


How to integrate a platform as a service with configuration management tools?

A configuration management tool is not required to manage a PAAS. For example, when one has created a VM on Azure then it is possible to export the configuration snippet as powershell or as yaml and use yaml to deploy and manage systems.

In my opinion one should restrict configuration management tools like puppet or ansible to manage configuration files and installing additional packages and yaml to manage the PAAS as I prefer KISS.

If one would like to manage different PASS then one could consider to use terraform.


Clarification about Operational Support System

The Wikipedia definition of Operational Support System (OSS) makes it sound like a term of art from telecommunication companies.  I haven't seen this come up in systems administration over many years of working in this area.  The top vendors in the OSS space are Ericsson, Nokia, and Huawei.  I never see those guys trying to help me manage cloud deployments, so I don't feel there's much overlap with what Ansible Tower does.
Ansible Tower provides a GUI for working with the anisble configuration management tool.  While this provides similar functionality to what it seems an Operational Support System to be, I doubt Ansible Tower is being used as an OSS by anyone.  It wasn't mentioned in any of the product comparisons I googled for OSS vendors.
Tools similar to Ansible Tower would be in categories like deployment tools or infrastructure visualization.  A few examples of these:

rundeck
DeployHub
Foreman

There's also overlap with CI/CD tools.


As per my experience, it is not a good idea because the availability of a pod is not directly related to how fast the container is up, a lot of other things affects the time the pod is ready and the time kubernetes sends traffic to that new pod.

You need to think about how your scalability strategy is trying to adapt to your actual traffic and the different health checks change that readiness time.

In Kubernete you have the livenessProbe and radynessProbe used to check two different thinks on the lifecycle of the pod, because of that you need to review the lifecycle of a pod in kubernetes and if your app can adapt to it.

https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/

In case you are worried about the number of requests you can handle per second my advice is to have a pool of pods ready to response a good amount of it and take metrics to know better how your app responds to a high demanding traffic context and understand how the loadbalancing of kubenetes work and can affect your app, with that try to scale in a more smart way.


VSTS Azure for DevOps: Unit test checking mechanism on pull request

If you have the option, Github offers Actions which allow you to run workflows on different events - one of them being push.

If Github is not an option, I would recommend creating a build where deployment of the artifacts only happens if the build is done on master.

On your feature branches, on checkin, a build will happen which will pull the latest code, compile, and then run unit tests, security scanning, and code coverage (if applicable). Since this is a feature branch, the build ends there.

If it is on master, it then performs the steps to package/deploy the artifacts.

In Branches > Master > Branch Policies (in the ellipses menu), add a Build Validation.

I recommend marking it required.

If you aren't worried about merge conflicts causing a rebuild, you can make it expire immediately upon master changing (requiring a rebuild, which will fail on merge conflicts).

If you are a large team where you might frequently step on one another, you can make it never expire and leave it up to the developer to fix manually before they are able to merge.


Kafka from client to server vs. "router" in the DC

Kafka doesn't care how you use it; it does nothing except transporting your arbitrary messages for you. Yes, you can use it that way, and actually I do not see anything specific enough about it to make it necessary to be mentioned in the docs.

Obviously you need to take a  little care regarding security (SSL or SASL...) but aside from that, there should be no particular problem.


RBAC to secure Azure Table Storage access

For now there is no support for accessing Azure Table Storage with Managed Identities.

Supported services are described here:

Services that support managed identities for Azure resources

According to documentation, Azure Table Storage allows you to authenticate only by using account key. So SAS authentication is not solution here either.

Possible solution could be like this:

Create Azure Key Vault and store account key as a secret Assign
permission to this key vault to your Managed Identity Use Managed
Identity to fetch the key from key vault in a runtime
Use the key to Azure Table Storage

As additional step you can create separate Storage Account to store only tables. With this approach you should be able to restrict access to tables service properly.


How do you reuse a specific volume in Kubernetes?

Yes, the PersistentVolumeClaim is the workhorse here. You can evict pods, delete deployments, etc., to your heart's content and so long as you don't delete the PVC, your data will be there.

You don't need to use StatefulSet for this.  From the docs:

Unlike a Deployment, a StatefulSet maintains a sticky identity for each of their Pods. These pods are created from the same spec, but are not interchangeable: each has a persistent identifier that it maintains across any rescheduling.

https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/

You might want that anyway, but it does nothing for keeping your PV/PVCs around.


I have a similar setup.

For feature/* and develop, I use something like 18.08-SNAPSHOT in my pom.
When we're ready to start release hardening, I'll cut a release branch (release/18.08) and change the version in the pom/s to 18.08.

I'll also bump the version on develop to 18.09-SNAPSHOT, for example.

From there, bugfix and finalise the release, merge to master when appropriate.

We're not properly CI/CD yet, but the issue I bumped in to with release builds was that I still needed unique builds in Artifactory.  So, at build time (for release branches) Jenkins will change the version in the pom to something like 18.08-326 (326 being build number).  It won't commit this change, but the artefact that gets built and pushed to Artifactory will be unique (foobar-18.08-326.war)


How do I upload my zipped lambda code into AWS Lambda?

It's kind of hidden away and not very intuitive.

Within the AWS Lamba admin: Create Function > Author from Scratch.
Within the Code entry type dropdown there is a Upload a .ZIP file option.


How to build windows image on azure dev ops hosted agent

EDIT
Microsoft added a new hosted agent: Hosted Windows 2019 with VS 2019.
So you should use this image to build for windows-1809 or windows-2019
Source : https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops&tabs=yaml#use-a-microsoft-hosted-agent

Old answer:
FWIW, I ended up creating a windows server 2019 vm on azure to act as a build agent.
Could not manage to build 2019 image in any of the hosted agent pools.


How do I run opendistro elasticsearch in docker on HTTP

in your docker-compose.yml you need change this parametres:
environment:
ELASTICSEARCH_URL: https://odfe-node1:9200
ELASTICSEARCH_HOSTS: https://odfe-node1:9200
For this:
environment:
ELASTICSEARCH_URL: http://odfe-node1:9200
ELASTICSEARCH_HOSTS: http://odfe-node1:9200
And after in your file config for elasticsearch put this:
https-enabled = false
Try and enjoy ;)


Is it possible to use a list with 'file.exists' in Salt?

Yes - there are at least two options available for managing files in the manner you describe. The first such way is to manage the entire directory using file.directory:

/opt/apiv2:
  file.directory:
    - user: root
    - group: root
    - dir_mode: 755
    - file_mode: 644
    - recurse:
      - user
      - group
      - mode

The second way can manage a manifest of several files at many paths is to use the source_hash feature of file.managed:

This allows you to provide a source tar file and a file containing md5sum hashes:

apiv2-0.7.3.tar.gz:
  file.managed:
    - name: /tmp/apiv2-0.7.3.tar.gz
    - source: salt:///apiv2/distrib/apiv2-0.7.3.tar.gz
    - source_hash: salt:///apiv2/distrib/manifest-0.7.3.hash

You would then create a text file on your salt server /srv/salt/apiv2/distrib/manifest-0.7.3.hash with contents similar to:

37b51d194a7513e45b56f6524f2d51f2    /opt/apiv2/apiv2.properties
acbd18db4cc2f85cedef654fccc4a4d8    /opt/apiv2/repo/bootstrap.jar
73feffa4b7f6bb68e44cf984c85f6e88    /opt/apiv1/apiv1.properties

Your might even be able to use templating,

apiv2-0.7.3.tar.gz:
  file.managed:
    - name: /tmp/apiv2-0.7.3.tar.gz
    - source: salt:///apiv2/distrib/apiv2-0.7.3.tar.gz
    - source_hash: salt:///apiv2/distrib/manifest-0.7.3.hash
    - template: jinja

however Salt Stack might want to push down the file each time because after the templates are rendered, the hash will have changed, so the creators might just have elected to throw an error if you try these two features together. In short, Your Mileage May VaryTM


Trigger cicd pipeline after merge request accept on master, Gitlab

https://docs.gitlab.com/ee/ci/yaml/

https://gitlab.com/gitlab-org/gitlab-ce/issues/31296#note_34944784

integration-testing:
  stage: test
  only:
    - master
  script:
    - ./gradlew clean build asciidoctor

Another option is to create a webhook

https://gitlab.scm.webanywhere.co.uk/help/user/project/integrations/webhooks.md

If the "Merge request events" is clicked one could trigger a pipeline if a Pull Request gets updated, created or merged.


How should I separate micro service modules for the message part(posts, reply, private) of a social network?

From my point of view, you can design microservices just like classes in an application. That is, you are looking for maximum cohesion within a service. In other words, all stuff inside one microservice should work together to achieve exactly one well-defined responsibility, and no other microservice should have any content for the same responsibility.

Furthermore, you want to minimize coupling. That is, mainly, that each microservice should have a very well defined interface, and should be easily replacable in the grand scheme of things. This also means making use of concepts like dependency injection and such.

So as long as you have these two aspects under control - cohesion and coupling - you are fine. The exact separation of concerns (more coarse or more fine-grained) is not that important and will likely differ between different applications.

There are obvious concerns here - for example, it would not make sense to put "Logging" and "GUI" in the same component (ask yourself questions like this: "Would anything else than a GUI use the Logging?" or "Does GUI need to know how Logging is implemented?" or "Does the GUI need to be implemented by the same team that does Logging?" and so on). Another frequent example would be handling of data imports - does that belong into the GUI service just because users usually upload their Excel files through a GUI? Obviously not - you may wish to upload Excel via a REST API or by pulling them from another server in a batch job, etc.

So, in your example, you best sit down and define the responsibilities for every component. Make sure there are as little overlaps as possible. This will usually lead to merging of components you thought to be different; or splitting up of components which turn out to have a well-defined border.


This is what I would to better handle the base docker images lifecycle:

Create a git repo to save your Dockerfile + configs
Create a Dockerfile with alpine + all the configs/libraries you want
Add your php.ini + php-cli.ini configs
Have a Jenkins job to start on each push to that repo:

Build your docker image
Tag it incrementally
Push to your registry

Use that base image to create your final docker image


kubernetes statefulset - communicate addresses of other pods when pod launches?

Yes you can by doing a DNS SRV lookup. Here's an example:

$ kubectl run -it srvlookup --image=tutum/dnsutils --rm --restart=Never -- dig SRV jira-headless.default.svc.cluster.local

; <<>> DiG 9.9.5-3ubuntu0.2-Ubuntu <<>> SRV jira-headless.default.svc.cluster.local
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 56846
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 2

;; QUESTION SECTION:
;jira-headless.default.svc.cluster.local. IN SRV

;; ANSWER SECTION:
jira-headless.default.svc.cluster.local. 30 IN SRV 10 50 0 jira-0.jira-headless.default.svc.cluster.local.
jira-headless.default.svc.cluster.local. 30 IN SRV 10 50 0 jira-1.jira-headless.default.svc.cluster.local.

;; ADDITIONAL SECTION:
jira-0.jira-headless.default.svc.cluster.local. 30 IN A 10.244.11.152
jira-1.jira-headless.default.svc.cluster.local. 30 IN A 10.244.0.188

;; Query time: 6 msec
;; SERVER: 10.233.0.3#53(10.233.0.3)
;; WHEN: Fri Jun 28 00:18:14 UTC 2019
;; MSG SIZE  rcvd: 143

In the example above I have a headless Service called jira-headless. If you pay attention to ADDITIONAL SECTION: you'll notice it provides the SRV record in the following format:

_service._proto.name. TTL class SRV priority weight port target.

Which maps to the Pods (along with their IPs) backing the Service:

jira-0.jira-headless.default.svc.cluster.local. 30 IN A 10.244.11.152
jira-1.jira-headless.default.svc.cluster.local. 30 IN A 10.244.0.188


Error with declaring volumes in Docker-compose, 'no such file or directory, open '/app/package.json'

As mentioned in the post/question, it was an issue with volumes, my PC's firewall was blocking the Docker from accessing the drives in my local machine, I reset the credentials in 'Docker for Desktop' app in the shared drives section and everything was back working once again :). Thank you everyone, in fact BMitch was suggesting the same solution, but I was a completely beginner and I did not comprehend it well. Thanks everyone for your responses.


Someone said that to each question, there are always three answers. So here's your third answer.

Install docker on the dev machine, and build images with it.

But:

The images should not contain your source code or anything that changes during the usual edit-build-test-run cycle. The images only need to contain the basic stuff that does the build/test/run parts.

That is, if you were to develop a Maven application, your image/container would contain the mvn command with all its depdendencies. A second image/container would contain your wildfly  or whatever you are using as an application server.

If you were to develop in ruby on rails, you would have one image/container which contains everything to be able to run bundle exec rake cucumber, another image/container which contains everything to be able to run rails s or rails c.

The images will be recreated seldomly, for example if you want to upgrade the application server, and they will certainly not be docker build't on each run by the dev. Both images would not contain anything that belongs to your application. No source code, no .war file, no precompiled assets or application-specific gems for the ruby application. These would be mounted as volume when starting the containers, and point directly to whatever local directory the dev is currently working with.

This way, it is guaranteed that your dev uses the exact same base environment as your CI or production. New devs can start building/testing/running instantly and never need to install anything to their host machine. They can use the editor of their choice (if they wish to use something like Eclipse which has lots of magic and wants to compile the stuff itself, then they are free to do so of course - as long as they do at least a cursory check whether everything still works in the docker variant, before pushing).

And if you wish to deploy your application as a fully self-contained docker image, finally, then you are free to do so. Your CI server can build those images. It should use the smaller images used by your devs as base image, obviously.

This gives you a nice medium ground which should cover everything.


AWS ECS Fargate, auto scaling and load balancer

Load balancing will route the requests across tasks while autoscaling will increase or decrease the number of tasks based on the traffic.
starting new tasks when cpu threshold exceed is autoscaling, this can be setup with cloudwatch events.
you can load balance services based on rules and then route them to target groups. read here


How can DevOps help to improve Software Escrow procedures?

A very interesting question. On the assumption that the goal of a Software Escrow process is to allow for a 3rd Party to take over or nominate an additional party to fulfil the responsibility of the software vendor, I would suggest the following elements of a DevOps operating model that would support software escrow:

Infrastructure-as-code - effectively documenting through an executable specification of dependant infrastructure, stored and versioned in source control provides the environments that the source code is developed in.  Unlike static documentation in text files because it is executed on a regular basis by the software vendor to build their own environments it does not go out of date of suffer from "bit rot".  This will always hold the most value when the entire development pipeline is built and maintained in source control applying Infrastructure-as-code principals.
Continuous Integration - the goal of continuous integration is to execute a set of steps that integrate the solution on a regular basis, ideally upon each change.  Typically this means upon check-in and push to a central repository a set of tests are executed that validates the process.  From a software escrow perspective I would expect this to also push a working version to a secondary "backup" repository owned and operated by the 3rd parties.  It is important to note that this does need to be both legally and financially "unlinked" from the software vendor.
Continuous Deployment - the goal of continuous deployment is to deliver software in a working state on a frequent basis.  In this sense the 3rd party is just another deployment target to deploy the outputs to, albeit probably not actively spinning up infrastructure on the 3rd parties fabric.

Some other considerations to bring into the equation are:

the move from static documentation to infrastructure as code significantly reduces the toil involved in updating documentation that describes how to install, configure and recover software.
don't forget secrets management, such as X.509 Certificates, symmetric keys, passwords and licence keys, these could be stored in source control however this does have it's own drawbacks.

From a tools perspective this will depend very much upon the development environment, I don't believe there are any tools specific to software escrow however:

.NET Development

GitHub or BitBucket for source control
AppVeyor, TeamCity or Jenkins for the build and test elements of the pipeline.
Ansible, Puppet or Chef for the configuration management elements of the pipeline.
ProGet for the package management elements of the pipeline
Octopus Deploy for the deployment aspects of the pipeline

Java Development, and to some extend Open Source platforms

CircleCI, TravisCI, TeamCity or Jenkins for the build and test elements of the pipeline.
Ansible, Puppet or Chef for the configuration management elements of the pipeline.
JFrog Artifactory or Sonartype Nexus for the package management elements of the pipeline
Capistrano for the deployment aspects of the pipeline

At the end of the day if you are able to employ Infrastructure-as-code, Continuous Integration and Continuous Deployment principles to a software package it can be used to fulfil your obligations under a Software Escrow contract.


What are best practices for implementing and using Elastic in an organization?

One best practice would be to log everything to the same location. Your "Current idea" suggests that you'd be running the Elastic stack locally, which is great for a local development environment, but for anything outside that you want to be shipping the logs from all your microservices to the same location. This enables you to do correlation between logs, and have a "single pane of glass" approach to your logging.

To achieve that, you're going to have to run it on a server or use a managed service such as Elastic Cloud or AWS Elasticsearch. You can install the components through package managers e.g. yum or apt, https://www.elastic.co/start, or it's also possible to run it in Docker. The stack-docker repository is great for running locally, but it's not suitable for any sizeable deployment due to a lack of scalability.

The endpoint you ship to will depend on what method/protocol you're using to ship logs, for example Nginx running on an AWS EC2 instance might use Filebeat and ship to a load balancer that sits in front of Elasticsearch, or if you're using Docker you might use the gelf Docker logdriver to ship your logs to Logstash.

Some more best practices:

Log all your messages in JSON, with a focus on objects/fields rather than long string messages.
Exceptions should be actionable, anything else can be a warning.
Avoid multiline logs.
Use correlation IDs. If your app is passed a correlation ID it should include it in every message triggered by that call, otherwise it should generate one. The correlation ID should be included in calls to other microservices.
Err on the side of logging more rather than less, you never know when you may need that bit of data, or what trends you might find by analysing an otherwise inconspicuous piece of data.
Separate your environments logically so you can have different retention periods and archiving on your logs.


How to solve 'upstream connect error or disconnect/reset before headers. reset reason: connection termination' in Istio?

I encountered the same error but with a different issue. The Service port needed a name added to it. https://github.com/istio/istio/issues/19966. And they need to follow the format (protocol-suffix) https://istio.io/docs/ops/deployment/requirements/

ports:
    - name: https # Use http or https
       protocol: TCP
       port: 8080
       targetPort: 8080


If you're interested in only backing up configs, try installing the SCM Sync Configuration Plugin. This plugin syncs with a repo each time you change either a job or the Jenkins configs, but does not keep builds.

SCM Sync Configuration Jenkins/Hudson plugin is aimed at 2 main features :
Keep sync'ed your config.xml (and other ressources) jenkins/hudson files with a SCM repository
Track changes (and author) made on every file with commit messages

As a bonus you can set it to take a commit message each time you change a job, for easier restoration if necessary.


How do I specify a database name for a SQL DB Deploy task?

It seems like Database Name is linked as a Process Parameter.

Under the "SQL Deployment" process, you should see the variable there in an editable state. You can either fill in the value there, or you can unlink and fill in the value in the "SQL DB Deploy" task.


What are off-the-shelf options for creating a microservice that consults a REST interface and shows the outcome in a graph?

I am currently looking into http://www.chartjs.org/. It looks like that this is able to create graphs. I will add more to this answer if I have created a graph using this library.


Bash Tokenization Errors on Jenkins JNLP Slave

I cannot say this is a the best answer so I do hope someone can provide a better clarification.

It seems the default shell for sh() in Jenkins is Bourne.  By updating the sh() with the correct shell this allows the below usage:

sh'''#!/bin/bash
    IFS='/'
    branch=($1)
    # Use it.
'''

This doesn't resolve the the redirection error.  Also I would think by updating the default shell in the image with chsh -s /bin/bash I wouldn't need to set it but I still do.


DMAIC - Define, Measure, Analyse, Improve, Control in the DevOps world?

Yes, DMAIC makes sense. If processes like CD need to be improved it is important to have a baseline. Therefore the Measure step is useful. The team investigates the status of process X, e.g. CD by documenting the current status.


How to update all Bitbucket Repository Issues at once?

At the moment it seems not be possible to update all issues at once using the API and therefore this issue has been created.


Passing variables between scripts in a Jenkins pipeline

For Declarative Pipeline:

This question already has a highly-upvoted answer on StackOverflow:

  If you want to use a file (since a script is the thing generating the value you need):

pipeline {
  agent { label 'docker' }
  stages {
    stage('one') {
      steps {
        sh 'echo hotness > myfile.txt'
        script {
          // trim removes leading and trailing whitespace from the string
          myVar = readFile('myfile.txt').trim()
        }
        echo "${myVar}" // prints 'hotness'
      }
    }
    stage('two') {
      steps {
        echo "${myVar}" // prints 'hotness'
      }
    }
    // this stage is skipped due to the when expression, so nothing is printed
    stage('three') {
      when {
        expression { myVar != 'hotness' }
      }
      steps {
        echo "three: ${myVar}"
      }
    }
  }
}

A second solution would be to write out output from your script directly or with writeFile and then later read it back in with readFile.  However, this comes with the significant caveat that both the reading and writing must take place on the same node, so you can't use this to send data across nodes.

You can also can archive files centrally with archiveArtifact and retrieve them with copyArtifact (and then read in the file with readFile).

Finally, there are some answers to a near-identical question on StackOverflow that offer alternative solutions.

However, if you can, I would suggest using Scripted Pipeline instead, which allows a much simpler solution:

def String myVar

stage('my-first-stage') {
  myVar = sh(script: 'my-command', returnStdout: true)
}

stage('my-second-stage') {
  sh("my-other-command --var='${myVar}'")
}

(I also already answered this question for Scripted Pipeline over on ServerFault.)

It looks like some of these solutions have already been proposed in the comments, but you are concerned that they are not elegant enough.  I'm not really sure what kind of elegance you're looking for, because I can't think of any way to simplify these answers in Jenkins-Pipeline-land.


How to approach dynamic environments per Git feature branch workflow?

What you're after is referred to as "Review Apps" by Gitlab and Heroku. It relies on a CI/CD system that runs pipelines on every branch, and exposes the branch name to the build (usually as an environment variable). You can then use the branch name to namespace the resources created by your Infrastructure as Code.

You have mentioned Docker so I will use Kubernetes as an example, but this approach can be applied to just about any deployment method.

Normally when deploying a Docker image, you would name your deployment after the app, e.g.:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
# ...

Using either some kind of preprocessor (Jinja, envsubst, etc.) or a tool like Helm we can pass in some parameters to make the resource include the branch name during the pipeline:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app-${BRANCH_NAME}
# ...

This way, branches do not deploy on top of each other. The resources they deploy are isolated to the branch.

For an API or web app, you'll want some kind of DNS name and load balancer. Kubernetes handles this quite easily with Ingress and DNS controllers, so I'll use CloudFormation as an example this time.

AWSTemplateFormatVersion: '2010-09-09'
Parameters:
  Branch:
    Type: String
  Environment:
    Type: String
  Zone:
    Type: String

Resources:
# ALB resources omitted for brevity...

  Route53:
    Type: AWS::Route53::RecordSet
    Properties:
      Name: !Sub
        - "my-app-${Branch}.${Environment}.${Zone}"
      HostedZoneName: !Ref Zone
      Type: A
      AliasTarget:
        HostedZoneId: !GetAtt ALB.CanonicalHostedZoneID
        DNSName: !GetAtt ALB.DNSName

Here we create a DNS record that is very specific to the branch. The DNS record resolves to a load balancer which should point to the containers deployed by the pipeline for this branch. The stack name is also specific to the branch, so pipelines on other branches will not affect it. The end result is that each feature branch will have a dedicated URL (https://my-app-my-feature.dev.example.com) that your developers can use to test and showcase to stakeholders.

  Should we create 6 new servers/containers if we only make changes in application A?

No. This isn't specifically related to Review Apps, but microservices in general should be self-contained: decoupled from each other so that they can be deployed in isolation. Removing dependencies and making small changes will increase your development/deployment velocity.

Considerations

Cleanup

What happens when you merge or delete your branch? Without setting up some kind of cleanup, those containers and DNS records will sit around forever. You need some kind of cleanup. Something that just runs periodically and deletes resources associated to old branches is pretty simple to write, or you could take webhooks from your git server and delete those resources on an event.

Databases

Some resources may not warrant getting deployed per-branch, although this should be the exception rather than the norm. You probably don't want to deploy a new database every time someone makes a branch (perhaps unless you have a good schema management tool and DB population scripts). You can have all deployments connect back to the same DB. Ensure that any changes you make to the schema are backwards compatible at least one version behind.


Docker Compose V3 app container can't access couchdb container

Your mistake is that you are did not register the app service on the app-net network. To allow containers to connect with one another, they have to share a network and then use the image name as the domain name. Docker provides a network internal DNS service.

version: '3'
services:
  couchdb:
    image: couchdb:latest
    restart: always
    volumes:
      - couchdb-data-volume:/usr/local/var/lib/couchdb
    env_file: .env.docker
    ports:
      - "12345:5986"
    networks:
      app-net:

  app:
    build: .
    image: app
    container_name: app
    restart: always
    depends_on:
      - couchdb
    env_file: .env.docker
    networks:
      - app-net

volumes:
  couchdb-data-volume:

networks:
  app-net:

The URL from the app to the couchdb service would be http://couchdb:5986. To test this, open bash inside the app service with docker exec -it app. This depends on the container_name setting.


How can I enable active tracing by default for an AWS Lambda function using Terraform?

This looks like it was added to Terraform back in May 2017, you enable it with the tracing_config block:

tracing_config {
  mode = "Active"
}

You will also need to make sure that Terraform has a minimum of Write access to X-ray to enable the above to work:

data "aws_iam_policy" "aws_xray_write_only_access" {
  arn = "arn:aws:iam::aws:policy/AWSXrayWriteOnlyAccess"
}


Continuously getting locked out from EC2 instance. No supported authentication methods available

Although i have not found out what leads to this. But i am finally able to login into my instance by following these steps:
1. As you are logged in create a user in the instance and give it a password.

$ adduser user1
$ passwd user1

2. Enable password authentication in /etc/sshd_config file.
3. Login with the new user.

Interestingly, this user does not gets locked out as the centos user.


Upload data into docker container from docker volume

You can mount a volume e.g. at /app/artifacts then set up your application to place the artifacts on that folder.

docker run -d \
-v artifacts-vol:/app/artifacts \
artifact-creator:latest

then after that, you can mount the same volume to a separate container and specify on the new container to read the artifacts from the volume location

docker run -d \
-v artifacts-vol:/app/artifacts \
artifact-reader:latest


docker-compose multiple services in a loop

Not in pure Docker Compose.  If you have a plain-text templating engine you like (Mustache, Jinja, m4, ...) you can ask it to do this for you.

You tagged this as shell-script, and Compose can take the YAML configuration on stdin, so in principle one option is to write a shell script, and then pipe that into docker-compose:

#!/bin/sh

buildComposeYaml() {
  cat <<HEADER
version: '3'
services:
HEADER
  for i in $(seq 3); do
    cat <<BLOCK
  s$i:
    build: ../www$i
    ports:
      - "500$i:9000"
BLOCK
  done
}

buildComposeYaml | docker-compose -f- "$@"

(Note that YAML is indentation-sensitive, and the script is mixing indentation for the script itself, the YAML embedded in the script, and the end-of-heredoc markers; so this will work for a simple tool but it wouldn't be my long-term choice.)


How to skip Build without creating Drop and not triggering Release in Azure DevOps

Try and add a Stage condition to only run that stage for a specific branch.
condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/master'))

stages:
- stage: A
             .....

- stage: B
  **condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/master'))**
  jobs:
  - job: B1
    steps:
      - script: echo Hello Stage B!
      - script: echo $(isMain)

Reference : https://docs.microsoft.com/en-us/azure/devops/pipelines/process/conditions?view=azure-devops&tabs=yaml
PS : Will refine the answer in future, currently we are still using GUI for releases and we are using this branch filter to control this scenario.


Unable to validate certificate chain: cert.pem in Zimbra mail server

This error occurrs because you do not add the root CA in your chain.pem file:

for solutions add below chain in your chain.pem file:

-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVow
PzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMRcwFQYDVQQD
Ew5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmTrE4O
rz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEq
OLl5CjH9UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9b
xiqKqy69cK3FCxolkHRyxXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw
7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40dutolucbY38EVAjqr2m7xPi71XAicPNaD
aeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNV
HQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQMA0GCSqG
SIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69
ikugdB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXr
AvHRAosZy5Q6XkjEGB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZz
R8srzJmwN0jP41ZL9c8PDHIyh8bwRLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5
JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo
Ob8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ
-----END CERTIFICATE-----

OR you can use below script for install Let's Encrypt SSL in your Zimbra server:

#!/bin/bash

# SSL certificate installation in Zimbra
# with SSL certificate provided by Let's Encrypt (letsencrypt.org)
# Author: Subhash (serverkaka.com)

# Check if running as root
if [ "$(id -u)" != "0" ]; then
   echo "This script must be run as root" 1>&2
   exit 1
fi

read -p 'letsencrypt_email [xx@xx.xx]: ' letsencrypt_email
read -p 'mail_server_url [xx.xx.xx]: ' mail_server_url

# Check All variable have a value
if [ -z $mail_server_url ] || [ -z $letsencrypt_email ]
then
      echo run script again please insert all value. do not miss any value
else

# Installation start
# Stop the jetty or nginx service at Zimbra level
su - zimbra -c 'zmproxyctl stop'
su - zimbra -c 'zmmailboxdctl stop'

# Install git and letsencrypt
cd /opt/
apt-get install git
git clone https://github.com/letsencrypt/letsencrypt
cd letsencrypt

# Get SSL certificate
./letsencrypt-auto certonly --standalone --non-interactive --agree-tos --email $letsencrypt_email -d $mail_server_url --hsts
cd /etc/letsencrypt/live/$mail_server_url
cat <<EOF >>chain.pem
-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVow
PzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMRcwFQYDVQQD
Ew5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmTrE4O
rz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEq
OLl5CjH9UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9b
xiqKqy69cK3FCxolkHRyxXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw
7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40dutolucbY38EVAjqr2m7xPi71XAicPNaD
aeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNV
HQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQMA0GCSqG
SIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69
ikugdB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXr
AvHRAosZy5Q6XkjEGB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZz
R8srzJmwN0jP41ZL9c8PDHIyh8bwRLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5
JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo
Ob8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ
-----END CERTIFICATE-----
EOF

# Verify commercial certificate
mkdir /opt/zimbra/ssl/letsencrypt
cp /etc/letsencrypt/live/$mail_server_url/* /opt/zimbra/ssl/letsencrypt/
chown zimbra:zimbra /opt/zimbra/ssl/letsencrypt/*
ls -la /opt/zimbra/ssl/letsencrypt/
su - zimbra -c 'cd /opt/zimbra/ssl/letsencrypt/ && /opt/zimbra/bin/zmcertmgr verifycrt comm privkey.pem cert.pem chain.pem'

# Deploy the new Let's Encrypt SSL certificate
cp -a /opt/zimbra/ssl/zimbra /opt/zimbra/ssl/zimbra.$(date "+%Y%m%d")
cp /opt/zimbra/ssl/letsencrypt/privkey.pem /opt/zimbra/ssl/zimbra/commercial/commercial.key
sudo chown zimbra:zimbra /opt/zimbra/ssl/zimbra/commercial/commercial.key
su - zimbra -c 'cd /opt/zimbra/ssl/letsencrypt/ && /opt/zimbra/bin/zmcertmgr deploycrt comm cert.pem chain.pem'

# Restart Zimbra
su - zimbra -c 'zmcontrol restart'

# setting auto https redirect
cd /opt && touch https-redirect.sh && chown zimbra:zimbra https-redirect.sh && chmod +x https-redirect.sh
cat <<EOF >>/opt/https-redirect.sh
zmprov ms $mail_server_url zimbraReverseProxyMailMode redirect
EOF
su - zimbra -c '/opt/https-redirect.sh'
rm /opt/https-redirect.sh
if

read this link for more details: https://github.com/SubhashPatel/Install-Zimbra-mail-server-ubuntu/blob/master/configure-ssl-zimbra.sh


Git repo setup for multiple dot net project

Only put a library in a separate repo when it is used in two different repos. If that is not the case then put the code in the same repo. Otherwise one could run into the anti-pattern nanoservices.


How to find the constraint in a software development shop?

To identify a constraint in any process is a relatively simple task. Work moves from a person to person throughout the organization and it will simply pile up in front of a constraint. You can look for people or teams with the highest number of blocking unresolved issues for example. It depends on how you track the flow of work, but if you do track it, then identifying constraint is usually not too difficult. It helps to know the value chains for your organization and simply follow along them.

Constraints can be of three types:

People: A particular skill or knowledge unique to an individual.
Policy: A written or unwritten policy or process, which limits the work either by increasing latency or reducing throughput.
Equipment: Particular computer system can limit the speed at which you can pass work through it or the number of people that can use it at the same time.

In regards to software, the Software Delivery Process is often a constraint and thus Continuous Integration / Continuous Delivery is such a powerful way to elevate this constraint and subordinating the organization to this constraint is often needed to increase the throughput of work. By automation in Software Delivery you remove slow people and often costly policy obstacles and approvals. Equipment in form of Computer System upgrades is a relatively small cost comparatively to the other two.

QA is another example of possible constraint in the process. Manual quality controls are work intensive and even automated testing can be constraint by equipment, in case the software is to run on expensive hardware. But the most common constraint in QA is policy, which tends to snowball with ever problem. The latency of the QA process and the small throughput can lead to larger units of work being put through, which can push back the cost into code integration.


AWS: Application vs Classic Load Balancer

1) you are using 2 classic ELBs instead of 1 ALB
so costs savings can be a factor

2) using a layer7 loadbalancer vs a layer4 gives many features and capabilities not possible otherwise

you can see a comparrison here:
https://aws.amazon.com/elasticloadbalancing/details/#compare

3) yes that wildcard would fit all your needs (sub1/sub2.domain.com)


CloudWatch trigger for a range of GuardDuty severities

I used this post to get our Guard Duty alerts working, thanks!  But a recent change at AWS caused us to stop getting any alerts.  Turns out we needed to add the integer values along with floating point numbers. We used a CLI command like the following to do that:

aws events put-rule --name Test --event-pattern "{\"source\":[\"aws.guardduty\"],\"detail-type\":[\"GuardDuty Finding\"],\"detail\":{\"severity\":[7.0,7.1,7.2,7.3,7.4,7.5,7.6,7.7,7.8,7.9,8.0,8.1,8.2,8.3,8.4,8.5,8.6,8.7,8.8,8.9,7,8]}}"

Note the "7" and "8" at the end.  Alerts are working again.


I think this is because your ECS service was created before tagging was available for ECS services.

See this blog post by AWS on how to migrate to the new ARN format. https://aws.amazon.com/blogs/compute/migrating-your-amazon-ecs-deployment-to-the-new-arn-and-resource-id-format-2/


Azure DevOps Build and Release pipeline artifact name variable

I needed to add:

variables:
- group: front_end-storage

At the top of my pipeline for the pipeline to pick up on the variables and authorize the group.


As documented here, ansible does not support telnet as connection plugin. This means you cannot use ansible to connect to a remote machine via telnet and execute any of the ansible modules.

However, you can use connection: local for local connection(run ansible against the machine that is running it) and send telnet commands from the machine where ansible runs.

Example playbook will look something like that:

---
- hosts: localhost
  connection: local
  gather_facts: false

  tasks:
    - name: my first telnet task
      telnet:
        username: user
        password: pass
        command:
          - my command


Availability/Reproducibility of Ubuntu Minimal as Docker image

The answer to the question "is there a Dockerfile for that, as well?"

https://github.com/tianon/docker-brew-ubuntu-core/tree/dist-amd64

Note their git branches contain the actual release build based on architecture.


How to have multiple log streams in docker

I'm still looking for a merging/splitting approach myself, but meanwhile this approach recommended by Kubernetes documentation seems like a sound solution: Use a sidecar container for each of your separate logs.

A "sidecar" is any docker container that you use alongside another docker container to work with it in some way. In this case for each of your three logs you would have a separate container that scans or tails the logs and outputs to stdout.

This way each of your log-sidecar-containers has its own docker-log from its own stdout. Being separate like this, you can use standard docker (and kubernetes, etc) practices for separating or aggregating. Here's what the Kubernetes page has to say:

  This approach allows you to separate several log streams from different parts of your application, some of which can lack support for writing to stdout or stderr. The logic behind redirecting logs is minimal, so it’s hardly a significant overhead. Additionally, because stdout and stderr are handled by the kubelet, you can use built-in tools like kubectl logs.

The "separate log streams" stem from the built-in tagging that docker applies to logs from different containers, described in the docker documentation here:

  The tag log option specifies how to format a tag that identifies the
  container’s log messages. By default, the system uses the first 12
  characters of the container id. To override this behavior, specify a
  tag option


These are presumably proprietary tools for EC2 and GCE respectively. The need for predictive VM autoscaling is pretty strong for Virtual Machines since provisioning time can be measured in the 10's of minutes depending on the configuration.
Conversely, application spin-up time in Kubernetes is typically measured in seconds. Predictive scaling for the Kubernetes HPA might have limited practical benefit.
Now using cloud provider predictive autoscaling to scale out the VMs running a Kubernetes cluster based upon ML could be pretty interesting.


Mobile CI: Versioning and Tracking Builds in JIRA

The answer will vary depend on the CI/CD tooling you are using.

If you are planning on using Atlassian's CI/CD tool Bamboo, you could use the Bamboo variable ${bamboo.buildNumber} which ties the build number of Bamboo to your version number.  You can have a task that will adjust the version based on this variable.  Bamboo
 integrates with Jira so that you can see the builds and deployments directly within Jira.  There is also Git so you could see the commits that went into the deployment.  This effectively would automate the second requirement you listed.

There are plugins available for Jira that will integrate with other CI/CD toolsets (e.g. Jenkins, CircleCI), and the idea would be the same: let the CI/CD server automate the versioning for you.


Does Spinnaker's rosco use HashiCorp's packer or is it another tool?

From spinnaker own documentation

  Rosco is the bakery. It is used to produce machine images (e.g. GCE
  images, AWS AMIs, Azure VM images). It presently wraps packer, but
  will shortly be expanded to support additional mechanisms for
  producing images.

So yes, rosco uses HashiCorp's packer.

The main reason to use packer is that you just need to change the provider used to create a docker image or an AWS ami or an Azure image and keep your building step identical.


What you want can be done. I've seen it done with GNU screen when Tmux wasn't around yet. However, I would not consider it good style to do it.

There are better ways to do this. Daemons are simply "normal" processes being run in a particular way. That particular way usually means to fork "twice" (so that the parent/child relationship gets broken) and plugging stdout, stdin and stderr to some file (or /dev/null if no logging is desired).

There are tools aplenty out there to achieve what you want while also ensuring that there is a policy in place if the process unexpectedly goes down. After all a service should be available as close to 100% of the time as possible.

Read up on:

daemontools
supervisor
daemonize
runit

Arguably your system will already have one of these or a very similar tool installed. But you don't give enough information for me to provide more detail.


Ansible ad-hoc command to reboot servers in serial with a pause

@Argyle explained the --fork 1 trick to achieve a serial run.

I suggest you have a look at the reboot module that might be better suited than running a command directly.

A quick example that will reboot all your nodes one by one waiting for each to be fully booted before moving to the next one.

ansible -i inventory nodes --forks 1 -m reboot

Pass needed parameters to the module depending on your requirements with the -a option.


Should I implement Devops on a small project. Is it too big for it?

The question you need to answer is "would I find it entertaining to set up a CI/CD environment if not to learn from it then just for the fun of it?".
You may not need it at this point in your project or it may benefit you. That I can't tell from the limited information. Linux started as a personal project, too. I set up something similar for my own pet project and I've gotta admit I feel a little bit proud every time I compile and deploy with a single click.

Should you implement it? That's for you to decide.


Jenkins libraries configuration in Jenkinsfile

You should look at Global Shared Libraries, which will allow you to reuse logic across different Jenkinsfiles.

Rather than depending on the environment, it's probably safer to explicitly pass parameters and returns values from the functions, which will make their behavior more explicit. Another side benefit is that you will be able to unit test them using


What are the best practices for anti-virus / anti-malware when using containers?

This is an area where you will need to talk directly to and potentially help educate your auditors.  What you have to remember is they are responsible for ensuring that adequate controls are applied, not specifically that "Production Servers have Endpoint Protection"; Anti-virus and Anti-malware are among some of the most basic controls applied to technology.

As a bit of a starter for ten, you could consider the following:

Secure Software Development Lifecycle (S-SDLC): In traditional environments, developer and administrator workstations would have Anti-virus and Anti-malware products installed, therefore any code that a developer creates and an administrator deploys will have been subject to routine automatic scans.  It's easy to forget that the same controls need to be applied to our automated pipelines, one option would be to scan both source code and artefacts with ClamAV or a commercial antivirus product.

Trivy - not anti-virus but does offer vulnerability scanning so controls against malware entering via an upstream 3rd party.
Anchore
Clair

One of the best practices for container development is reducing the attack surface area by maintaining a hardened and minimal base image.  For example, removing any non-essential components such as shells, network utilities, and text editors.  Utilising a pre-hardened image such as the CIS Images and applying
If you manage your own Kubernetes masters and nodes then you should consider the attack vectors against that host use endpoint protection to mitigate, although if you use Hosted Kubernetes, i.e. Google Kubernetes Engine or Azure Kubernetes Service, you need to jump through some hoops to get it working.  As you've probably already seen there is some basic advice on what to avoid, and you need to understand the limitations of the scanning engine - namely it can't see into containers.

Docker Anti-virus Exclusions - remember Kubernetes is using ContainerD
Kubernetes ClamAV
Sysdig Falco
Carbon Black

Equally, masters and nodes should be leveraging SELinux as a Mandatory Access Control security mechanism and generally have the relevant CIS or STIG benchmarks applied to them.
There are a small but growing number of "hot" container endpoint protection tools, it's fairly nascent so explore your options carefully:

Dagda
OpenSCAP
Sysdig Falco

Integrity Checking is very important as whilst it isn't anti-malware per se, it is a component part of most anti-malware packages so might just give you the signal you need to know that something is wrong:

images should be signed, and Kubernetes should be configured to only use images signed by a trusted party.
hosts should absolutely employ File Integrity Monitoring with something like twistlock.
equally, file integrity monitoring within the container itself can be valuable.

Finally, remember "Infrastructure as Code" if you can destroy all of your infrastructures and rebuild it from scratch automatically it makes it significantly easier to recover from a compromise.  This is a tough sell with an auditor.

If you implement everything above, you will quickly find you run into information overload - this is where a Security Information and Event Management (SIEM) tool can help bring everything together in a single pane of glass.


Running consul in the background using Chef

The usual way to do that is to use a service manager (systemd or supervisord) to handle the service lifecycle (keeping the consul program running and restarting it if it fails for exemple) there's cookbooks existing to help creating those service units.

The cheap alternative if you count on chef running periodically and handling that is to use nohup which is a linux program made exactly for this.


Let docker containers work with each other

What would be the best way to share ports among the containers?

You don't really "share" ports between containers. Instead, you want to create network and attach each container to that network.  Each container gets a network alias (essentially a hostname) that you can use to hit the service.

$ docker network create foo
$ docker run --network=foo --network-alias=mysql -p 3306:3306 my-mysql:latest
$ docker run --network=foo --network-alias=restapi -p 5000:5000 --link mysql my-restapi:latest
$ docker run --network=foo --network-alias=react -p 3000:3000 --link restapi my-reactapp:latest

So the myrestapi container can access the mysql container using the hostname mysql and port 3306 and your react app can hit the myrestapi container using the hostname restapi.  The containers will also be accessible from your localhost so you can hit your react app with a browser without additional setup

In docker-compose.yml (which I don't use a ton so someone may have a better solution), it would look something like this:

networks:
  foo:
services:
  mariadb:
    container_name: mysql # this works for your alias so long as you don't have another container called mysql
    image: mariadb:latest
    networks: ["foo"]
    ports:
    - 3306:3306

The other services would follow suit.

As for "priming" the database, that's a little trickier. If the DB has to be running, you would need to replace the docker container's entrypoint to start the server, run the mysql command to import the data, then your script would just spin forever.

Alternately, you could script your container start-up and after it starts, import the data that way.


Alarms management with Elasticsearch and Kibana

I have no experience with it myself, but Elastalert (http://elastalert.readthedocs.io/en/latest/elastalert.html) sounds like what you need (for the alarms/threshold part)

But besides that, it sounds like your interested in monitoring. In my opinion ELK is not a classic monitoring tool (more: data collection, data processing and data visualization). You should take a look at Icinga2 (https://www.icinga.com/products/icinga-2/) which can can be configured highly dynamical and is compatible with nagios plugins


Can we do zero-downtime kubernetes deployment with a single pod?

Is there anyway to configure K8s to actually bring up a new pod, direct new traffic to it, and only then shutdown the old pod?

This is the default behavior. MaxSurge is number of additional pods that runs during upgrade.
What you also have to check is how the load balancer is handling traffic, does it have capability to detect that you have deployed a new Pod, e.g. is it Kubernetes aware? See Container native load balancing on Google Cloud, it looks like you should have a ReadinessProbe on your Deployment.

Actually I realized what's happening is that the pod is getting traffic routed to it before the app is actually completely ready.

This is the default behavior, if you define a ReadinessProbe for your Deployment. As you write, you can also add a StartupProbe but the ReadinessProbe is most important.


Should I use filebeat or an app-logging module with lumberjack capabilities to collect my logs

Both are common and comes with their own pro and cons.

It depends if you can accept loosing logs if logstash is dead and your app restart and on the other hand if you can write logs to disk and consume place there.

Those are the basic points to choose for one or the other, you may end up with more caveats in term of IO/caching/etc.

There's no 'best' option, there's the best option for your organization/teams which we can't answer for you as we don't know your environment.


Using Anaconda instead of pip+virtualenv

In my field (science) Anaconda is probably the most common Python distribution in use. conda is the package manager (just to be clear of the distinction).

It is difficult to find fault in conda when compared with pip+virtualenv.

The one point that I would make is that the Anaconda distribution is meant to be installed per user. If you have a Python environment that is shared across users there is no way of doing the equivalent of pip install --user with conda (but read on for a surprising solution...). Nor users can create conda environments.

So, if you are considering a single user installation go ahead and try Anaconda. It is worth a try

Note that you can use pip with Anaconda... they are not incompatible. In fact that can be quite useful in a shared installation.

I have setup a shared-user conda installation where users can install user-space packages with... pip install --user.

In a shared installation I do not see how users can create their own conda environments (conda has environments). And I bet conda+virtualenv would be a mess.

Note that conda is not Python tied: First you can have a conda installation with both Python 2 and 3 - and conda manages environments very well. But you can also have conda install Perl, R. In fact that is quite common. Our Perl interpreter is provided by Anaconda ;)

Also I suspect conda is probably stronger in Linux and Mac OSes. Windows is less well supported. For other operating systems, there are probably no packages (or they will be limited).


The idea is to put all our sensitive data [...]

The meaning of "all" in this sentence should be analyzed very carefully before implementing the solution that you plan.

Ansible vault is a very useful tool, but it should be used only to store secrets that are:

Specifically needed for the ansible deployments
Easily made useless to owners that should become unaware of them, but that may illegitimately "remember" them (typically off-boarded employees)

The second point is critical.

Many people, and potentially the whole DevOps team, will have access to the ansible vault password and therefore all the secrets.

Therefore, for all the secrets stored in the vault, a condition should hold for which a person or machine with unauthorized access to them should be incapable of making any use of them if so is desired.

In concrete terms, if you use ansible to deploy a database and its users, you can store the passwords in the vault, but you will have to be very careful (and most likely consider another solution) if that service will be available from the Internet and without the need for any VPN authentication!

Users (DevOps) exposed to the secret, should be incapable of using "remembered" passwords if one security barrier is imposed on them (e.g., VPN access revoked). In addition to this, access to the source code repository (where the vault is stored) should be revoked as well before passwords are changed.

Under these conditions, ansible vault is a very useful tool.

Trying to store a secret that could be used by any person or machine on the Internet in the vault would be instead a mistake (e.g., VPN credentials of users).

  Is there any other options, which is the best (and secure) way to store ansible-vault password

Under the conditions from the previous paragraph, I think that a good practice would be:

Store the vault password in an external secure vault (something like Vault from HashiCorp or any SaaS for credentials management)
Allow access to the external vault item to DevOps (they will need the password for testing) and the CI/CD system or ansible controller
Keep a convention to use secrets! You will not be able to review changes to the secrets and you will not be able to grep for ansible variables in secrets files! So be thorough since the beginning. A good convention is to name all variables stored in the ansible vault with a secret_ prefix. When you will see something like:

postgres.yml:

postgres_password: "{{ secret_postgres_password }}"

you will know that the value is stored in the ansible vault.


Terraform deletes AWS lambda layers instead of keeping the older versions

Terraform's job is to maintain one remote object per resource block, either by updating that same object over time or by replacing it with a new object that meets new requirements.
If you wish to gradually accumulate new objects over time then Terraform is not the correct tool for the job. Your need would be better served by an application deployment tool, such as the Serverless framework.


I ran into the same issue and found this docker-compose file that is able to start postgres and sonarqube.

https://github.com/SonarSource/docker-sonarqube/blob/master/recipes.md

version: "2"

services:
  sonarqube:
    image: sonarqube
    ports:
      - "9000:9000"
    networks:
      - sonarnet
    environment:
      - SONARQUBE_JDBC_URL=jdbc:postgresql://db:5432/sonar
    volumes:
      - sonarqube_conf:/opt/sonarqube/conf
      - sonarqube_data:/opt/sonarqube/data
      - sonarqube_extensions:/opt/sonarqube/extensions
      - sonarqube_bundled-plugins:/opt/sonarqube/lib/bundled-plugins

  db:
    image: postgres
    networks:
      - sonarnet
    environment:
      - POSTGRES_USER=sonar
      - POSTGRES_PASSWORD=sonar
    volumes:
      - postgresql:/var/lib/postgresql
      # This needs explicit mapping due to https://github.com/docker-library/postgres/blob/4e48e3228a30763913ece952c611e5e9b95c8759/Dockerfile.template#L52
      - postgresql_data:/var/lib/postgresql/data

networks:
  sonarnet:
    driver: bridge

volumes:
  sonarqube_conf:
  sonarqube_data:
  sonarqube_extensions:
  sonarqube_bundled-plugins:
  postgresql:
  postgresql_data:


How to perform continuous Integration using Jenkins?

Continuous integration means roughly what you said. Whenever something is committed, it is automatically built and tested, with a red/green light showing up at the end.

Continuous deployment takes this a step further and whenever a green light turns up, automatically deploys that.

Note that "tested" above can mean several steps: smoke tests (does it crash and burn right away), automated unit/feature/integration tests, and then, if you are not superbly confident, also maybe some final manual tests. Also, obviously, for CD, your test setup needs to be sufficiently conclusive so you can deploy with good confidence. You can also do the fully automatic deploy to internal testing/staging areas first, and then only do a production deploy at a later date.

To your questions:

  But in my organization there is no expertise who clearly tell us about how CI tool actually used in real time.

You need to figure that out for yourself as it depends very much on the persons, processes and software you are building/deploying. At first, just build the tools, work with them, and see what happens. If everything works splendidly, continue along the path you laid out yourself; if you hit problems, solve them as they come up.

  So we are not aware of the scope and expansions of the Automation project in terms of CI tool.

The more automation, the merrier. Automation not only means shaving off a few minutes here and there (but this factor also pays of hugely surprisingly fast), but also that everything that is automated, is codified in some script or tool; which means you are much less dependent on individual persons (who may become ill, overloaded, or leave the company) and scale much better if your team grows.

  manage the release using Jenkins ?

I view Jenkins one step "lower" than managing releases. In "pure" CI/CD, the final stage, there would be no releases anymore as every commit would directly and in real-time end up on production anyways. At the other end of the spectrum, you have something like 3 releases per year, each one of them late by 2 months ;). Every project will be somewhere along that spectrum, as dictated by circumstances (not the least of which are your stakeholders and the available money - running a very fast CI/CD is not free).

  the terms Jenkins pipeline confused me what it is and why it used for?

"Pipeline" is just a fancy word for saying that you commit - build - test - deploy. It's just figuratively speaking. The imagery is that you drop your commit into the pipe at the one end, and on the other end you get a running production server.

  So after committing the new branch Jenkins will trigger the build for that branch. It is the right approach ?

Sure, it's worthwhile to have every commit on every branch tested, by the default, if you have the resources for that. It lets you know exactly which build broke everything. Coupled with very small commits, this is a quite powerful tool.

Another useful trigger is to not build every commit, but have Jenkins look at the repository every 15 minutes or so, and build if there is one more more new commits. This may be more realistic when your test suites take longer to run, and if you do not have unlimited amounts of test environments that you can spin up and down regularly.

(By the way, if you are completely new to all of this, including "agile" and branching, then maybe it would be good for you to pick up an existing branching strategy, for example Gitflow, so you don't complicate things too much by (re-)inventing too much stuff at the same time.)


Is there an overview of current resource usage in a GCP region?

This page indicates something about regions and zones, but does not show any resource usage. I think that the resource usage of a zone is not visible.


Jenkins Pipeline downstream job; exit unless it hasn't run in the last x hours

To synthesize Alex and Steve Johnson's answers, you could put this at the top of your downstream job:

lastBuild = currentBuild.getPreviousBuild()
delay = 2

if Date(lastBuild.timestamp) >= new Date().minusHours(delay) {
    currentBuild.result = 'SUCCESS'
    echo "Less than ${delay} hours since the last build. Job will not run."
    return
}


Yaml linting failling when jinja is used

My bad, I was searching using the wrong keyword, I just found out that ignoring a file or dir is quite easy, you just need to add an gitignore style block to your yamllint config file, as :

ignore: |
  *globalParamName.sls
  *platform.sls

See the official doc here.


How securely to keep google cloud service account key?

I would recommend you use their Google Cloud Secrets Engine. Documentation is here: https://www.vaultproject.io/docs/secrets/gcp/index.html

Managing such sensitive secrets by any means other than a Vault service is dangerous.


OpenShift fails to deploy jenkins - connection refused

Have you set Jenkins ssh key to have access to source?
Another point is to validate if the SSH port is open. According to error can be one of these things


What opensource solutions can be used to implement anomaly detection based on Riemann metrics?

I can think of 3 possibility (Not exactly Riemann metrics based, but around anomaly detection):

Outdated but the most efficient in anomaly detectionI know: Shyline + Oculus from Etsy Both tools available from etsy's github as Skyline and oculus, check the network graph for more up to date forks. Best one I found is earthgecko/skyline with its doc
Graphite and checking that the Standard deviation fits in the Holtz Winter's Confidence bands
ELK stack in version 5 with timelion which, as above,  allow to compute a standard deviation and alert if it goes out of confidence bands with the proper license.


Is scrum or kanban really useful for SRE teams?

We don't use Agile for the DevOps group ourselves, but we do integrate with the normal Scrum Teams. When something is needed by the team from DevOps, such as optimizing the build server, the related team puts a PBI in their backlog with a 'DevOps' label. Our lead has a custom dashboard in Jira with all issues labeled 'DevOps'. They work with the Scrum Master to get prioritization and then one of the DevOps Engineers is tasked with being an ad-hoc member of that Scrum Team for the life of that issue. This helps us prioritize work based on our "customer's" priorities, ties our effort to the sprint, and allows us to get "credit" for the work we're doing.

Prior to this, we used Kanban on Jira just for a todo list. Unfortunately, we would sometimes have to stop working on something we had planned to work on because one of the teams needed something. Now, unless it's an emergency, they basically request DevOps resources (people/time) via their backlog and the Scrum Master communicating that need to our Lead.


The “easiest” approaches to deploying such app is probably using a Platform as a Service (PaaS). Here are some of them to get you started, pick any, really:

Python Anywhere (https://www.pythonanywhere.com/), focused on Python apps (hence Django as well)
Heroku (https://heroku.com/)
Clever-Cloud (https://www.clever-cloud.com/)
Google App Engine (https://cloud.google.com/appengine)
AWS ElasticBeanstalk (https://aws.amazon.com/fr/elasticbeanstalk/)
Azure AppServices (https://azure.microsoft.com/services/app-service/)
RedHat OpenShift (https://www.openshift.com/), open-source, available self-hosted or as a service

To me the first three are the easiest, but it all eventually boils down to personal preference.


How to persuade developers to start using feature flag toggles?

Feature toggles are a common practice in high-velocity development because they de-couple development from release.  Dev teams can "soft-release" a new feature to production, in a disabled state.  This allows the feature to be released any time.  If the feature is dependent on other work or preparation, it doesn't have to wait for a major release to go to production.

As far as "convincing" developers to use them, that's an exercise in making the case for the freedom it offers. My experience is that it's not a tough sell to developers.  It's management that tends to be reluctant to try new things.  Try this:

Find a feature toggle framework.  Management/the business may be more
amenable to trying something supported by a common system
Start small.  Introduce the toggle system on a trial basis for a feature that will demonstrate its utility.
Demonstrate the toggle's ability to do A/B testing.  Enable the toggle for a subset of your web farm, then gather metrics on behavior.  Tiny differences in page layout have been demonstrated to have huge effects on revenue for retail applications (e.g. Ebay, Amazon)


Does Jenkins job create Jenkinsfile? On execution

No, Jenkins file is not created at execution.
Actually Jenkins file is a suite of all steps or activity which you want to build as a job or a groovy code.

Jenkins file come from code repository, if you added file in your project.

Below is a Jenkins file example:

pipeline {
    agent any

    stages {
        stage('Build') {
            steps {
                echo 'Building..'
            }
        }
        stage('Test') {
            steps {
                echo 'Testing..'
            }
        }
        stage('Deploy') {
            steps {
                echo 'Deploying....'
            }
        }
    }
}

for more reference watch this video: https://www.youtube.com/watch?v=KZY4Rwsewro


In a Database Per Service model - should i include the database in the service container?

Keep the DB containers separate. The concept of microservices and containers is to allow each part to be independently updated without introducing changes and outages to the rest of the application stack. Containers should also be running a single app per container to allow error detection and log gathering from that application. Each container is simply an isolated process, so unlike a VM, there's little overhead on the host to split each of these out.


Separate SQL Server or just separate SQL Database for test and production?

Go with a separate logical server:

It is going to cost the same because you are charged on the tier of the database not the database server.
By providing separate servers you provide isolation between the two databases protecting one from the other.
Unless you use contained users your users will be shared between Test and Production, if you happen to use the same user for both databases a change of the password in production would also require the same change in Test.

You might want to use the same server for a database if:

You are creating a very large number of Databases across many Dev/Test environments as there is a limit to the total number of SQL Servers.
You have two databases that need to be deployed as part of the same ARM template.


How to automate initial server configuration?

What you are trying to accomplish is referred to as infrastructure as code.
Which tools you select will depend on where you are operating, and what your business needs are.  If you are operating in AWS exclusively, you can skip 3rd party solutions and focus on their CloudFormation service.  Terraform (infrastructure) and Ansible (configuration) are popular cloud agnostic tools if you are operating On-Prem or across multiple cloud providers.


Use dynamic values for parameter-store in aws buildspec.yml for CodeBuild

I faced the same problem while using different SSH key pairs for different deployment environment. Usually I used to create different ssm variables per environment and then changed the name of ssm per git branch in buildspec.yml. That worked perfectly for me. For example, for deploying to stage environment i used the git branch "dev" which had following buidlspec.yml config
env:
  parameter-store:
    SECRET_KEY_BASE: "DEV_SECRET_KEY_BASE"

for "release" environment it was,
env:
  parameter-store:
    SECRET_KEY_BASE: "RELEASE_SECRET_KEY_BASE"

make sure you resolve merge conflicts between inter git branches before you apply.


Recommended Design - Communication among Jenkins + Vault + App

My choice would be design B.

It's true that the availability of the Vault would be crucial, but Jenkins' availability would similarly be crucial in design A. So from this perspective the decision would normally be driven by the comparison of the two availability figures, if available. Based on its functionality I suspect that Jenkins is likely to be a more complex tool than Vault (but I don't have any hard data supporting it). If that's correct then Jenkins would normally have a lower availability/reliability score.

A slightly different aspect of the dependency on Jenkins in design A isn't exactly appealing: Jenkins' job normally ends once it successfully deploys a certain version of the app, it isn't really designed to be around to "support" that app version during its expected lifetime (that includes restarting the app). Other tools are better suited to perform app monitoring and management in general. The app's independence from Jenkins that design B brings is, IMHO, an advantage from this perspective.


How can I make artifacts available to the QA team with a user-friendly UI?

It seems to me that you're looking for an artifact store such as Nexus or Artifactory.
Both solutions have Jenkins plugins which will simplify the process of uploading your artifacts to them.
You can find an example as to how to upload artifacts from your Jenkins pipeline to Nexus in this link and an example for Artifactory in this link.
Both solutions are very common and relatively easy to manage and use and are known by the community for a long time so you can find many tutorials that explain how they can be used and benefit you.
Then, you just need to allow your QA team read access to the relevant artifacts repo and they can download whatever version of the artifacts they need.


Cross account access from ec2 to other accounts

Supposing the scenario with two accounts A & B the explanatory steps should be:

In account A, I created a role (e.g RoleForB) to trust account B, and attach to the before created role a IAM policy to allow it to perform some read operations in account A. e.g ReadOnlyAccess
In account B, I created a role (e.g AssumeRoleInA) and attach a policy to allow it to assume the role that is created in account A.
In account B Associate to your EC2 instance ec2-profile the IAM role (AssumeRoleInA) created in step 2.
In account B login into this EC2 instance to assume the role in Account A using the command aws sts assume-role --role-arn "arn:aws:iam::Account_A_ID:role/RoleForB" --role-session-name "EC2FromB".
In account B EC2 terminal when the command is step 4. finished, you can see the access key ID, secret access key, and session token from wherever you've routed it, in our case stdout either manually or by using a script. You can then assign these values to environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN)

So Let’s check the configurations mentioned above step by step but with some mode detail:

As before presented in account A, it builds the trust to account B by creating the role named RoleForB and attaching ReadOnlyAccess permission to it.

{
    "Version": "2012-10-17",
    "Statement": {
        "Effect": "Allow",
        "Principal": {"AWS": "arn:aws:iam::Account_B_ID:root"},
        "Action": "sts:AssumeRole"
    }
}

In account B, create a role named AssumeRoleInA then attach the corresponding policy to allow it to assume the role named RoleForB in account A.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": [
        "arn:aws:iam::Account_A_ID:role/RoleForB"
      ]
    }
  ]
}

In account B, create a new EC2 instance (if it does not exists yet), and associate it's ec2-profile with the IAM role named AssumeRoleInA.

{
    "Version": "2012-10-17",
    "Statement": {
        "Effect": "Allow",
        "Principal": {"Service": "ec2.amazonaws.com"},
        "Action": "sts:AssumeRole"
    }
}

In account B login into this EC2 instance to assume the role in Account A using the command:

aws sts assume-role --role-arn "arn:aws:iam::Account_A_ID:role/RoleForB" --role-session-name "EC2FromB"`

eg:

jenkins@bb-jenkins-vault:~$ aws sts assume-role --role-arn arn:aws:iam::521111111144:role/DeployMaster --role-session-name "project-dev-jenkins-deploy"
{
    "AssumedRoleUser": {
        "AssumedRoleId": "AROAJBXGEHOQBXGEHOQ:project-dev-jenkins-deploy",
        "Arn": "arn:aws:sts::521111111144:assumed-role/DeployMaster/project-dev-jenkins-deploy"
    },
    "Credentials": {
        "SecretAccessKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
        "SessionToken": "FQoGZXIvYXCUm8iG6/zLdQ7foognvCDpxKP7cRJiZgc...CUm8iG6/zLdQ7foognvCDpxKP7c+OQF",
        "Expiration": "2019-03-29T15:41:02Z",
        "AccessKeyId": "AKIAI44QH8DHBEXAMPLE"
    }
}

In account B EC2 terminal when the command is step 4. finished, you can see the access key ID, secret access key, and session token from wherever you've routed it, in our case stdout either manually or by using a script. You can then assign these values to environment variables

$ export AWS_ACCESS_KEY_ID=AKIAI44QH8DHBEXAMPLE
$ export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
$ export AWS_SESSION_TOKEN=FQoGZXIvYXCUm8iG6/zLdQ...<remainder of security token>
$ aws ec2 describe-instances --region us-east-1

credit: https://stackoverflow.com/a/55420220/6657158


Way to Connect to all ec2-instances one by one

You can benefit from EC2 inventory script provided by Ansible - click
It will allow you to to connect to all ec2-instances, one by one (in serial: 1 mode)


How to manage Hashicorp Vault patching in AWS?

The canon solution is to use KMS auto-unseal: https://learn.hashicorp.com/vault/operations/ops-autounseal-aws-kms


How to generate a Maven.pom file (not pom.xml)?

Answer from here:

  Every jar needs to have a pom file describing it, you can just add
  something simple like this:

Every jar needs to have a pom file describing it, you can just add something simple like this:

http://maven.apache.org/maven-v4_0_0.xsd">
    4.0.0
    aGroup
    aArtifactId
    aVersion
    jar
    a Name


Why `docker stack` needs Docker to run in swarm mode? What are the implications when working on a single node?

Why it is mandatory to switch to swarm mode to run docker stack?

A stack and service are swarm mode objects by their very definition. The stack is a collection of services, and each service is a definition of a target state. Swarm mode takes those target states and manages containers to achieve that target state.

  And most important, what are the implications of switching to swarm mode when working on a single node?

Pretty minimal from the user perspective. There's another few ports opened and listening. If you try to directly manage containers (stop, rm, restart) that are managed by a service, you'll find that swarm mode will take steps to get you back to the service definition, which confuses some that keep finding new containers recreated when they delete all running containers directly. You just need to run the docker service commands to manage them instead.